Opinions expressed are solely my own, and do not express the views or opinions of my employer.

When I finally pieced together all the clues, I was shocked. If I was right, MongoDB was about to make what I would call the biggest mistake ever made in the history of database companies.

I work on an open source analytics tool that connects to NoSQL databases like MongoDB, so I spend my days rooting for these next-generation database vendors to succeed.

In fact, I just presented to a packed room at MongoDB Days Silicon Valley, making a case for companies to adopt the new database.

So when I uncovered a secret this destructive, I hit the panic button: on November 12th, 2015, I sent an email to Asya Kamsky, Lead Product Manager at MongoDB.

While polite, I made my opinion crystal clear: MongoDB is about to make a giant mistake, and should reconsider while there's still time.

I would never hear back from Asya — or anyone else about the matter. My earlier success in helping convince MongoDB to reverse course when they tried to monetize the wrong feature would not be repeated.

This is the story of what I discovered, how I pieced together the clues from press releases, YouTube videos, and source code scattered on Github, and how I ultimately failed to convince MongoDB to change course.

The story begins on June 1st 2015, at the annual MongoWorld conference in New York City.

MongoWorld 2015

SlamData, my new analytics startup, was sponsoring MongoWorld 2015, so I got a rare ticket to the VIP party the night before the conference.

Hosted at NASDAQ MarketWatch, in a beautiful space overlooking Times Square, I felt distinctly underdressed in my cargo pants and startup t-shirt. Fancy h'ordeuvres and alcohol flowed freely, and MongoDB's management team was out in full force.

I shook hands with MongoDB's new CEO, Dev ("Dave") Ittycheria, and offered him a few words of encouragement for the road ahead.

Only this year, Fidelity Investments slashed its valuation of MongoDB to 50% of what it was back in 2013 ($1.6B), downgrading the startup from "unicorn"  to "donkey".

It's been Dev's job to prove Fidelity and the rest of the naysayers wrong.

Dev inherited the company from Max Schireson (who famously resigned in 2014), and in his tenure, Dev has built out a new management team at MongoDB, with ripples felt across the company.

Though I only spoke with Dev for a few minutes, he seemed bright, friendly, and eager to learn about what my company was doing. He handed me his card and asked me to call him if I ever needed anything.

Next up was Eliot Horowitz, CTO and co-founder of MongoDB. I shook his hand, introduced myself, and delivered a 30 second pitch for my startup.

At the time, I thought my pitch must have been terrible, since Eliot seemed disinterested in everything I was saying. Turns out Eliot hates SQL and views analytics as a nuisance, so it's not surprising I bored him!

Eliot did catch the word "analytics", however, and dropped that tomorrow at the conference, MongoDB would have some news about the upcoming 3.2 release that I would find very interesting.

I pleaded for more details, but nope, that was strictly confidential. I'd find out the following day, along with the rest of the world.

I passed along the tip to my co-founder, Jeff Carr, and we shared a brief moment of panic. The big fear for our four-person, self-funded startup was that MongoDB would be announcing their own analytics tool for MongoDB, which could hurt our chances of raising money.

Much to our relief, we'd find out the following day that MongoDB's big announcement wasn't an analytics tool. Instead, it was a solution called MongoDB BI Connector, a headline feature of the upcoming 3.2 release.

The MongoDB 3.2 BI Connector

Eliot had the honor of announcing the BI connector. Of all the things he was announcing, Eliot seemed least interested in the connector, so it got barely more than a mention.

But details soon spread like wildfire thanks to an official press release, which contained this succinct summary:

MongoDB today announced a new connector for BI and visualization, which connects MongoDB to industry-standard business intelligence (BI) and data visualization tools. Designed to work with every SQL-compliant data analysis tool on the market, including Tableau, SAP Business Objects, Qlik and IBM Cognos Business Intelligence, the connector is currently in preview release and expected to become generally available in the fourth quarter of 2015.

According to the press release, the BI connector would allow any BI software in the world to interface with the MongoDB database.

News of the connector caught fire on Twitter, and the media went into a frenzy. The story was picked up by TechCrunch and many others. Every retelling added new embellishments, with Fortune even claiming the BI connector had actually been released at MongoWorld!

Given the nature of the announcement, the media hoopla was probably justified.

When Worlds Collide

MongoDB, like many other NoSQL databases, does not store relational data. It stores rich data structures that relational BI software cannot understand.

Kelly Stirman, VP of Strategy at MongoDB, explained the problem well:

“The thing that defines these apps as modern is rich data structures that don’t fit neatly into rows and columns of traditional databases."

A  connector that enabled any BI software in the world to do robust analytics on rich data structures, with no loss of analytic fidelity, would be giant news.

Had MongoDB really done the impossible? Had they developed a connector which satisfies all the requirements of NoSQL analytics, but exposes relational semantics on flat, uniform data, so legacy BI software can handle it?

A couple months earlier, I had chatted with Ron Avnur, VP of Products at MongoDB. Ron indicated that all of MongoDB's customers wanted analytics, but that they hadn't decided whether to build something in-house or work with a partner.

This meant that MongoDB had gone from nothing to magic in just a few months.

Pulling Back the Curtain

After the announcement, Jeff and I headed back to our sponsor booth, and Jeff asked me the most obvious question: "How did they go from nothing to a BI connector that works with all possible BI tools in just a couple months?!?"

I thought carefully about the question.

Among other problems that a BI connector would need to solve, it would have to be capable of efficiently executing SQL-like analytics on MongoDB. From my deep background in analytics, I knew that efficiently executing general-purpose analytics on modern databases like MongoDB is very challenging.

These databases support very rich data structures and their interfaces are designed for so-called operational use cases (not analytical use cases). The kind of technology that can leverage operational interfaces to run arbitrary analytics on rich data structures takes years to develop. It's not something you can crank out in two months.

So I gave Jeff my gut response: "They didn't create a new BI connector. It's impossible. Something else is going on here!"

I didn't know what, exactly. But in between shaking hands and handing out cards, I did some digging.

Tableau showed a demo of their software working with the MongoDB BI Connector, which piqued my curiosity. Tableau has set the standard for visual analytics on relational databases, and their forward-thinking big data team has been giving NoSQL some serious thought.

Thanks to their relationship with MongoDB, Tableau issued a press release to coincide with the MongoWorld announcement, which I found on their website.

I pored through this press release hoping to learn some new details. Burried deep inside, I discovered the faintest hint about what was going on:

MongoDB will soon announce beta availability of the connector, with general availability planned around the MongoDB 3.2 release late this year. During MongoDB’s beta, Tableau will be supporting the MongoDB connector on both Windows and Mac via our PostgreSQL driver.

These were the words that gave me my first clue: via our PostgreSQL driver. This implied, at a minimum, that MongoDB's BI Connector would speak the same "language" (wire protocol) as the PostgreSQL database.

That struck me as more than a little suspicious: was MongoDB actually re-implementing the entirety of the PostgreSQL wire protocol, including support for hundreds of PostgreSQL functions?

While possible, this seemed extremely unlikely.

I turned my gaze to Github, looking for open source projects that MongoDB might have leveraged. The conference Wifi was flaky, so I had to tether to my phone while I looked through dozens of repositories that mentioned both PostgreSQL and MongoDB.

Eventually, I found what I was looking for: mongoose_fdw, an open source repository forked by Asya Kamsky (whom I did not know at the time, but her profile mentioned she worked for MongoDB).

The repository contained a so-called Foreign Data Wrapper (FDW) for the PostgreSQL database. The FDW interface allows developers to plug in other data sources, so that PostgreSQL can pull the data out and execute SQL on the data (NoSQL data must be flattened, null-padded, and otherwise dumbed-down for this to work properly for BI tools).

"I think I know what's going on", I told Jeff. "For the prototype, it looks like they might be flattening out the data and using a different database to execute the SQL generated by the BI software."

"What database?" he shot back.

"PostgreSQL."

Jeff was speechless. He didn't say a word. But I could tell exactly what he was thinking, because I was thinking it too.

Shit. This is bad news for MongoDB. Really bad.

PostgreSQL: The MongoDB Killer

PostgreSQL is a popular open source relational database. So popular, in fact, it's currently neck-and-neck with MongoDB.

The database is fierce competition for MongoDB, primarily because it has acquired some of the features of MongoDB, including the ability to store, validate, manipulate, and index JSON documents. Third-party software even gives it the ability to scale horizontally (or should I say, humongously).

Every month or so, someone writes an article that recommends PostgreSQL over MongoDB. Often, the article goes viral and skyrockets to the top of hacker websites. A few of these articles are shown below:

• Goodbye MongoDB. Hello PostgreSQL
• Postgres Outperforms MongoDB and Ushers in New Developer Reality
• MongoDB is dead. Long live Postgresql :)
• Why You Should Never Use MongoDB
• SQL vs NoSQL KO. Postgres vs Mongo
• Why I Migrated Away from MongoDB
• Why you should never, ever, ever use MongoDB
• Is Postgres NoSQL Better than MongoDB?
• Bye Bye MongoDB. Guten Tag PostgreSQL

The largest company commercializing PostgreSQL is EnterpriseDB (though there are plenty of others, some older or just as active), which maintains a large repository of content on the official website arguing that PostgreSQL is a better NoSQL database than MongoDB.

Whatever your opinion on that point, one thing is clear: MongoDB and PostgreSQL are locked in a vicious, bloody battle for mind share among developers.

From Prototype to Production

As any engineer worth her salt will tell you, prototypes aren't for production.

Even if MongoDB was using PostgreSQL as a prototype BI connector, maybe some brilliant MongoDB engineers were locked in a room somewhere, working on a standalone production version.

Indeed, the way Tableau worded their press release even implied the dependency on the PostgreSQL driver might be temporary:

During MongoDB’s beta, Tableau will be supporting the MongoDB connector on both Windows and Mac via our PostgreSQL driver.

Perhaps, I thought, the 3.2 release of MongoDB would ship with the real deal: a BI connector that exposes the rich data structures that MongoDB supports (instead of flattening, null-padding, and throwing away  data), executes all queries 100% in-database, and has no dependencies on competing databases.

In July, more than a month after MongoWorld, I dropped by MongoDB's offices in Palo Alto during a business trip. And I was very encouraged by what I learned.

A Trip to MongoDB

By Palo Alto's standards, MongoDB's office is quite large.

I had seen the company's sign during previous trips to the Valley, but this was the first time I had a chance to go inside.

The week before, I was chatting with Asya Kamsky and Ron Anvur by email. We were discussing my company's open source work in executing advanced analytics on rich data structures directly inside MongoDB.

Since we happened to be in Palo Alto at the same time, Asya invited me over to chat over catered pizza and office soda.

Within the first few minutes, I could tell that Asya was smart, technical, and detail-oriented — exactly the traits you'd hope for in a product manager for a highly technical product like MongoDB.

I explained to Asya what my company was doing, and helped her get our open source software up and running on her machine so she could play with it. At some point, we started chatting about BI connectors for MongoDB, of which there were several in the market (Simba, DataDirect, CData, and others).

We both seemed to share the same view: that BI software needs to gain the ability to understand more complex data. The alternative, which involves dumbing down the data to fit the limitations of older BI software, means throwing away so much information, you lose the ability to solve key problems in NoSQL analytics.

Asya thought a BI connector for MongoDB should expose the native MongoDB data structures, such as arrays, without any flattening or transformations. This characteristic, which I have termed isomorphic data model, is one of the key requirements for a general-purpose NoSQL analytics, a topic I've written about extensively.

I was very encouraged that Asya had independently come to the same conclusion, and felt confident that MongoDB understood the problem. I thought the future of analytics for MongoDB looked very bright.

Unfortunately, I could not have been more wrong.

MongoDB: For Giant Ideas Mistakes

Delighted that MongoDB was on the right track, I paid little attention to the BI connector for the next couple of months, though I did exchange a few emails with Asya and Ron.

Heading into September, however, I encountered utter silence from the product team at MongoDB. After a few weeks of unreturned emails, I grew restless, and started poking around on my own.

I discovered that Asya had forked a project called Multicorn, which allows Python developers to write Foreign Data Wrappers for PostgreSQL.

Uh oh, I thought, MongoDB is back to its old tricks.

More digging turned up the holy grail: a new project called yam_fdw (Yet Another MongoDB Foreign Data Wrapper), a brand new FDW written in Python using Multicorn.

According to the commit log (which tracks changes to the repository), the project had been built recently, after my July meeting with Asya Kamsky. In other words, this was post-prototype development work!

The final nail in the coffin, which convinced me that MongoDB was planning on shipping the PostgreSQL database as their "BI connector", happened when someone forwarded me a video on YouTube, in which Asya demoed the connector.

Worded very cautiously, and omitting any incriminating information, the video nonetheless ended with this summary:

The BI Connector receives connections and can speak the same wire protocol that the Postgres database does, so if your reporting tool can connect via ODBC, we will have an ODBC driver that you will be able to use from your tool to the BI Connector.

At that point, I had zero doubt: the production version of the BI connector, to be shipped with MongoDB 3.2, was, in fact, the PostgreSQL database in disguise!

Most likely, the actual logic that sucked data out of MongoDB into PostgreSQL was a souped-up version of the Python-based Multicorn wrapper I had discovered earlier.

At this point, no one at MongoDB was returning emails, which to any sane person, would have been enough to call it quits.

Instead, I decided to give it one more try, at the MongoDB Days conference on December 2, just one week before the release of 3.2.

Eliot Horowitz was delivering a keynote, Asya Kamsky would be speaking, and Ron Avnur would probably attend. Possibly, even Dev himself might drop by.

That's when I'd have my best chance of convincing MongoDB to ditch the BI connector shenanigans.

MongoDB Days 2015, Silicon Valley

Thanks to the wonderful marketing team at MongoDB, and based on the success of a similar talk I gave in Seattle at a MongoDB road show, I had a 45 minute presentation at the MongoDB Days conference.

My official purpose at the conference was to deliver my talk on MongoDB-powered analytics, and make users aware of the open source software that my company develops.

But my personal agenda was quite different: convincing MongoDB to can the BI connector before the impending 3.2 release. Failing that lofty and most likely delusional goal, I wanted to confirm my suspicions about the connector.

On the day of the conference, I went out of my way to say hello to old and new faces at MongoDB. Regardless of how much I may disagree with certain product decisions, there are many amazing people at the company just trying to do their jobs.

I had gotten sick a few days earlier, but copious amounts of coffee kept me (mostly) awake. As the day progressed, I rehearsed my talk a few times, pacing the long corridors of the San Jose Convention Center.

When the afternoon rolled around, I was ready, and gave my talk to a packed room. I was excited about how many people were interested in the esoteric topic of visual analytics on MongoDB (clearly the space was growing).

After shaking hands and exchanging cards with some of the attendees, I went on the hunt for the MongoDB management team.

I first ran into Eliot Horowitz, moments before his keynote. We chatted kids and food, and I told him how things were going at my company.

The keynote started sharply at 5:10. Eliot talked about some of the features in 3.0, since a lot of companies are apparently stuck on older versions. He then proceeded to give a whirlwind tour of the features of MongoDB 3.2.

I wondered what Eliot would say about the BI connector. Would he even mention it?

Turns out, the BI connector was a leading feature of the keynote, having its own dedicated segment and even a whiz-bang demo.

The BI Connector

Eliot introduced the BI connector by loudly making the proclamation, "MongoDB has no native analytics tools."

I found that somewhat amusing, since I wrote a guest post for MongoDB titled Native Analytics for MongoDB with SlamData (Edit: MongoDB has taken down the blog post, but as of 15:30 MDT, it's still in the search index). SlamData is also a MongoDB partner and sponsored the MongoDB Days conference. 

Eliot seemed to stumble a bit when describing the purpose of the BI connector (getting actions from... actionable insights? Pesky analytics!). He looked relieved when he handed the presentation over to Asya Kamsky, who had prepared a nice demo for the event.

During the presentation, Asya seemed uncharacteristically nervous to me. She chose every word carefully, and left out all details about what the connector was,  only covering the non-incriminating parts of how it worked (such as its reliance on DRDL to define MongoDB schemas). Most of the presentation focused not on the BI connector, but on Tableau (which, of course, demos very well!).

All my feedback hadn't even slowed the BI connector down.

Pulling Out All the Stops

After the keynote, the swarm of conference attendees proceeded to the cocktail reception in the adjacent room. Attendees spent most of their time talking to other attendees, while MongoDB employees tended to congregate in bunches.

I saw Ron Avnur chatting with Dan Pasette, VP of Server Engineering, a few feet from the keg of Lagunitas IPA they were serving attendees.

Now was the time to act.

The 3.2 release was coming out in mere days. No one at MongoDB was returning emails. Eliot had just told the world there were no native analytics tools for MongoDB, and had positioned the BI connector as a revolution for NoSQL analytics.

With nothing to lose, I walked up to Ron, inserted myself into the conversation, and then began ranting against the BI connector in what was probably a two-minute, highly-animated monologue.

I told him I expected more from MongoDB than disguising the PostgreSQL database as the magical solution to MongoDB analytics. I told him that MongoDB should have demonstrated integrity and leadership, and shipped a solution that supports the rich data structures that MongoDB supports, pushes all computation into the database, and doesn't have any dependencies on a competing database.

Ron was stunned. He began to defend the BI connector's "pushdown" in vague terms, and I realized this was my chance to confirm my suspicions.

"Postgres foreign data wrappers support barely any pushdown," I stated matter-of-factly. "This is all the more true in the Multicorn wrapper you're using for the BI connector, which is based on an older Postgres and doesn't even support the full pushdown capabilities of the Postgres FDW."

Ron admitted defeat. "That's true," he said.

I pushed him to defend the decision. But he had no answer. I told him to pull the stop cord right now, before MongoDB released the "BI connector". When Ron shrugged off that possibility, I told him the whole thing was going to blow up in his face. "You might be right," he said, "But I have bigger things to worry about right now," possibly referring to the upcoming 3.2 release.

We had a beer together, the three of us. I pointed to Dan, "This guy's team has built a database that can actually do analytics. Why aren't you using it in the BI connector?" But it was no use. Ron wasn't budging.

We parted ways, agreeing to disagree.

I spotted Dev Ittycheria from across the room, and walked over to him. I complimented the work that the marketing department was doing, before moving on to critique product. I told Dev, "In my opinion, product is making some mistakes." He wanted to know more, so I gave him my spiel, which I had repeated often enough to know by heart. He told me to followup by email, and of course I did, but I never heard back.

After my conversation with Dev, it finally sunk in that I would not be able to change the course of MongoDB 3.2. It would ship with the BI connector, and there wasn't a single thing that I could do about it.

I was disappointed, but at the same time, I felt a huge wave of relief. I had talked to everyone I could. I had pulled out all the stops. I had given it my all.

As I left the cocktail reception, and headed back to my hotel, I couldn't help but speculate on why the company was making decisions that I so strongly opposed.

MongoDB: An Island of One

After much reflection, I now think that MongoDB's poor product decisions are caused by an inability to focus on the core database. This inability to focus is caused by an inability to cultivate a NoSQL ecosystem.

Relational databases rose to dominance, in part, because of the astounding ecosystem that grew around these databases.

This ecosystem gave birth to backup, replication, analytics, reporting, security, governance, and numerous other category-defining applications. Each depended on and contributed to the success of the others, creating network benefits and high switching costs that are proving troublesome for modern-day NoSQL vendors.

In contrast, there's virtually no ecosystem around MongoDB, and I'm not the only one to notice this fact.

Why isn't there an ecosystem around MongoDB?

My snarky answer is that because, if you are a MongoDB partner that provides native analytics for MongoDB, the CTO will get up on stage and say there are no tools that provide native analytics for MongoDB.

More objectively, however, I think the above is just a symptom. The actual problem is that the MongoDB partner program is totally broken.

The partner team at MongoDB reports directly to the Chief Revenue Officer (Carlos Delatorre), which implies the primary job of the partner team is to extract revenue from partners. This inherently skews partner activities towards large companies that have no vested interest in the success of the NoSQL ecosystem (indeed, many of them produce competing relational solutions).

Contrast that with small, NoSQL-centric companies like SlamData, Datos IO, and others. These companies succeed precisely in the case that NoSQL succeeds, and they provide functionality that's standard in the relational world, which NoSQL databases need to thrive in the Enterprise.

After being a partner for more than a year, I can tell you that almost no one in MongoDB knew about the existence of SlamData, despite the fact that SlamData acted as a powerful incentive for companies to choose MongoDB over other NoSQL databases (e.g. MarkLogic), and an enabler for companies considering the switch from relational technology (e.g. Oracle).

Despite the fact that partners try, MongoDB appears completely unconcerned about the joint revenue and sales opportunities presented by NoSQL-centric partners. No reseller agreements. No revenue sharing. No sales one-pagers. No cross-marketing. Nothing but a logo.

This means that organizationally, MongoDB ignores the NoSQL-centric partners who could most benefit them. Meanwhile, their largest customers and prospects keep demanding infrastructure common to the relational world, such as backup, replication, monitoring, analytics, data visualization, reporting, data governance, query analysis, and much more.

This incessant demand from larger companies, combined with the inability to cultivate an ecosystem, forms a toxic combination. It leads MongoDB product to try to create its own ecosystem by building all possible products!

Backup? Check. Replication? Check. Monitoring? Check. BI connectivity? Check. Data discovery? Check. Visual analytics? Check.

But a single NoSQL database vendor with finite resources cannot possibly build an ecosystem around itself to compete with the massive ecosystem around relational technology (it's far too expensive!). So this leads to distractions, like MongoDB Compass, and "sham" technology, like the BI connector.

What's the alternative? In my humble opinion, it's quite simple.

First, MongoDB should nurture a vibrant, venture-funded ecosystem of NoSQL-centric partners (not relational partners with deep pockets!). These partners should have deep domain expertise in their respective spaces, and all of them should succeed precisely in the case that MongoDB succeeds.

MongoDB sales reps and account managers should be empowered with partner-provided information that helps them overcome objections and reduce churn, and MongoDB should build this into a healthy revenue stream.

Second, with customer demand for related infrastructure satisfied by NoSQL-centric partners, MongoDB should focus both product and sales on the core database, which is how a database vendor should make money!

MongoDB should develop features that have significant value to Enterprise (such as ACID transactions, NVRAM storage engines, columnar storage engines, cross data center replication, etc.), and thoughtfully draw the line between Community and Enterprise. All in a way that gives developers the same capabilities across editions.

The goal should be for MongoDB to drive enough revenue off the database that product won't be tempted to invent an inferior ecosystem.

You be the judge, but I think it's pretty clear which is the winning strategy.

Bye-Bye, MongoDB

Clearly, I cannot get behind product decisions like shipping a competing relational database as the definitive answer to analytics on a post-relational database like MongoDB.

In my opinion, this decision is bad for the community, it's bad for customers, and it's bad for the emerging space of NoSQL analytics.

In addition, to the extent it's not done with full transparency, it's also bad for integrity, which is a pillar on which all companies should be founded (especially open source companies).

So with this post, I'm officially giving up.

No more frantic emails to MongoDB. No more monopolizing management at MongoDB cocktail parties. No more sharing my opinions in private with a company that doesn't even return emails.

Been there, done that, didn't work.

I'm also, obviously, blowing the whistle. By the time you're reading this, the whole world will know that the MongoDB 3.2 BI Connector is the PostgreSQL database, with some glue to flatten data, throw away bits and pieces, and suck out whatever's left into PostgreSQL.

What does all this mean for companies evaluating MongoDB?

That's your call, but personally, I'd say if you're in the market for a NoSQL database, you need legacy BI connectivity, and you're also considering PostgreSQL, you should probably just pick PostgreSQL.

After all, MongoDB's own answer to the problem of analytics on MongoDB is to pump the data out of MongoDB, flatten it out, and dump it into PostgreSQL. If your data is going to end up as flat relational data in PostgreSQL, why not start out there, too? Kill two birds with one stone!

At least you can count on the PostgreSQL community to innovate around NoSQL, which they've been doing for years. There's zero chance the community would package up the MongoDB database into a sham "PostgreSQL NoSQL" product, and call it a revolution in NoSQL database technology.

Which is, sadly, exactly what MongoDB has done in reverse.

------------------------------

The Shame photo taken by Grey World, copyright Grey World, and licensed under CC By 2.0.