Babelfish: the Elephant in the Room?

On December 1st, 2020, Amazon AWS announced Babelfish. Babelfish “adds an endpoint to PostgreSQL that understands the SQL Server wire protocol Tabular Data Stream (TDS), as well as commonly used T-SQL commands used by SQL Server. Support for T-SQL includes elements such as the SQL dialect, cursors, catalog views, data types, triggers, stored procedures, and functions”. Wow. SQL Server wire and application compatibility for PostgreSQL!

What this means is that Babelfish will be able to “impersonate” a SQL Server database. Applications may be able to run unchanged, believing that they are connecting to SQL Server, when they will actually be connecting to PostgreSQL-Babelfish.

Surely, compatibility will not be 100% at the beginning. But it will keep improving, and as long as it provides enough compatibility for a nice set of applications to run unchanged on Babelfish, it will open the door to migrations and replacing SQL Servers with Babelfish. This is very good news for the PostgreSQL Community!

Brief analysis on PostgreSQL popularity

PostgreSQL has been named (again) database of the year 2020. This award is given based on “DBMSs sorted by how much they managed to increase their popularity in 2020”, which means that PostgreSQL was the database that grew the most in popularity in 2020. But in absolute terms, PostgreSQL’s popularity is still way behind that of Oracle, MySQL and SQL Server. Let’s analyze db-engines popularity trend chart for the Top4 DBMS, on a linear scale (db-engines presents results on a logarithmic scale):

The trends for the last 8 years are clear: Oracle and SQL Server are constantly declining in popularity; MySQL is slightly declining; and PostgreSQL is clearly growing in popularity. But while PostgreSQL almost tripled in popularity in these eight years, it is still far behind the other three.

PostgreSQL became the database of 2020 because its popularity grew the most in the last year. The other three mentioned databases declined in popularity during 2020. If we assume the same rate of change in popularity will continue for the upcoming years, by 2025 PostgreSQL would still remain in the 4th place, albeit close to SQL Server. It won’t overtake SQL Server until 2026, and by 2030 PostgreSQL would still lag behind Oracle and MySQL.

How is this analysis related to Babelfish? Babelfish opens the door to new users, new markets, new opportunities. Babelfish may bump PostgreSQL’s popularity further, targeting use cases that either PostgreSQL is not able to reach today; or can only reach via complex technology migrations. Babelfish could be one of the many potential boosts that PostgreSQL needs in order to become a more universally used database.

To fork, or not to fork, that’s the question

In their announcement, AWS said that “We are open sourcing Babelfish in 2021. […] We are releasing Babelfish under the Apache 2.0 license. We invite others to become active in the project, and we will see it as a sign of success when developers outside of AWS become committers or maintainers. You can help by adding or extending Babelfish functionality, submitting feature requests, working on documentation, and contributing test cases”. They also mentioned the code will be published on GitHub.

At the time Babelfish will be published, it will likely require changes to PostgreSQL to enable Babelfish to be an extension. Some people may call that a “fork” of PostgreSQL, and others may call it a “development branch”. The difference between the two will only be clear over time. Note that PostgreSQL development model doesn’t use feature branches, and in my opinion it’s a great model –but this is an entirely different topic. It is really appreciated that AWS is releasing the code as open source, and under a permissive license (that should be compatible with PostgreSQL’s). If AWS developers work with the PostgreSQL Community to get the necessary changes merged into PostgreSQL core, then it would have been a development branch, and not a fork. This is something that the PostgreSQL Community and all parties involved need to figure out.

On January 25th, Amazon AWS made a first move. In an email to PostgreSQL’s hackers mailing list, Jan Wieck, a well-known Postgres-er, proposed to start discussing the implementation of protocol hooks. These hooks would enable to implement SQL Server’s protocol as an extension, rather than a fork.

Protocol hooks are, possibly, not the only hooks or modifications to PostgreSQL core that would be required to integrate the whole Babelfish project. With those changes to the core most of the Babelfish code could possibly be integrated in core as an extension(s). I cannot estimate the complexity of these core changes. But I believe that it would be a worthwhile effort, an effort that may further increase PostgreSQL outreach.

Only very recently (Feb 10th, 11th) some initial, very interesting, discussion around this proposal has started (including a very interesting offer to open source MySQL protocol compatibility for Postgres!). It’s understandable, it’s a very busy time for PostgreSQL hackers (the last Commitfest for feature inclusion into PostgreSQL 14 is ongoing). But Babelfish was already announced more than two months ago; and it could be published anytime. I believe we need to start a deeper conversation about Postgres-Babelfish integration sooner than later. And what is being discussed so far are mainly technical considerations around the integration of one of the possibly several integration points that may be required. I’d love to also have a strategic discussion, where the Community would address, from a leadership perspective, what would be the plans for integrating, or not, Babelfish. Is Babelfish the Elephant in the Room? Probably not anymore, but I anyway hope this post will help, at the very least, to spark the strategic discussion.

What is the alternative? What will happen if PostgreSQL would not implement such hooks or will not pursue understanding with AWS, for the common benefit, and help Babelfish and PostgreSQL cooperate and allow for code bases integration?

Under this scenario, AWS will probably have to keep Babelfish as a fork. For one, AWS already keeps Aurora as a fork, even though it’s an internal one. Given AWS’ well-known customer obsession and that AWS doesn’t kill services that they started offering, I think that if given no other chance they will keep Babelfish as a separate fork, getting its own share of features and contributors. Surely AWS knows that maintaining a fork is expensive. And I believe they have no intention to have it as a fork (otherwise, they won’t be publishing it as open source). So there will be serious intentions and efforts to merge it into PostgreSQL, for the benefit of all. But the recent Elastic case has demonstrated that AWS is committed to the open source software that is part of their managed services, and are willing to step up with a lot of resources when they are required to continue serving their customers. Apparently AWS has around 200 open job positions (!!) for developers working on Elasticsearch. Surely they can do the same for Babelfish, especially given that RDS Postgres/Aurora/Babelfish is probably a much larger business for them than Elastic.

On protocol hooks

Jan also argued that “Creating the necessary infrastructure in the postmaster and backend will open up more possibilities, that are not tied to our compatibility efforts. Possible use cases for wire protocol extensibility include the development of a completely new, not backwards compatible PostgreSQL protocol or extending the existing wire protocol”. I cannot agree more. This effort not only benefits the Babelfish integration; but also opens the door for new PostgreSQL protocols.

The current PostgreSQL protocol (v3) has been in use since PostgreSQL 7.4, released in 2003. It works well, and has spun the broadest possible set of drivers, tools and even compatible databases that use it (like CockroachDB, Crate.io or NoisePage, for example). But it also has some limitations and well-known problems. There is an entry in PostgreSQL “TODO” about proposed changes for an eventual v4 version of the protocol. I also participated in another “brain dump” on v4 proposed features. But despite much talk, v4 has not happened and there’s no ongoing effort to make it happen. v3 is to stay for long.

Why is that, why can’t the protocol evolve? Have a look at this thread, where a proposal to implement an HTTP protocol for PostgreSQL was made. Other than the proposal about HTTP itself –which has its own merits, and is a topic that I believe should definitely be discussed again–, the general sentiment was that any new protocol would have to provide all the features that the current protocol has, work for every use case, do not disrupt existing drivers or provide good means for driver rewrites; and do it significantly better than the current one.

The Innovator’s Dilemma

I don’t have an MBA, but this to me is a clear case of The Innovator’s Dilemma. PostgreSQL protocol v3 is the incumbent, and protocol v4 and/or other protocols like HTTP are the potential disruptive innovators. In what looks like a perfect match for Christensen’s book, a disruptive protocol innovation “would not initially satisfy the demands of even the high end of the market”. In other words, initial versions of these protocols should not target feature parity with the incumbent. They should rather focus on doing the basics, but much better, with a compelling higher value proposition.

Eventually, these protocols “will surpass sustaining technologies” and may end up replacing the current v3 protocol. This was very well explained by Prof. Clayton and is represented on his famous graph comparing the product performance of sustaining and disruptive technologies:

http://web.mit.edu/6.933/www/Fall2000/teradyne/clay.html

Protocol v3 should coexist with any other new protocol, be it v4, HTTP, or others. This approach would allow to develop new functionality and new protocol approaches with zero disruption to the current protocol, driver and technology ecosystem.

Now guess what’s needed to enable multiple protocols to co-exist: protocol hooks. The same pledge that Jan, from AWS, made. Let’s start the conversation. Do we want Babelfish to require a fork, work as an extension, or be fully integrated into PostgreSQL? Do we want to let Babelfish become “the MariaDB of PostgreSQL”? Our full answer is awaiting. Is there an Elephant in the Room?