There are many organizations where front/web-facing applications use MySQL and back end processing uses PostgreSQL®. Any system integration between these applications generally involves the replication—or duplication—of data from system to system. We recently blogged about pg_chameleon which can be used to replicate data from MySQL® to PostgreSQL. mysql_fdw can play a key role in eliminating the problem of replicating/duplicating data. In order to eliminate maintaining the same data physically in both Postgres and MySQL, we can use mysql_fdw. This allows PostgreSQL to access MySQL tables and to use them as if they are local tables in PostgreSQL. mysql_fdw can be used, too, with Percona Server for MySQL, our drop-in replacement for MySQL.
This post is to showcase how easy it is to set that up and get them working together. We will address a few points that we skipped while discussing FDWs in general in our previous post
Preparing MySQL for fdw connectivity
On the MySQL server-side, we need to set up a user to allow for access to MySQL from the PostgreSQL server-side. We recommend Percona Server for MySQL if you are setting it up for the first time.
1 | mysql> create user 'fdw_user'@'%' identified by 'Secret!123'; |
This user needs to have privileges on the tables which are to be presented as foreign tables in PostgreSQL.
1 2 3 4 5 | mysql> grant select,insert,update,delete on EMP to fdw_user@'%'; Query OK, 0 rows affected (0.00 sec) mysql> grant select,insert,update,delete on DEPT to fdw_user@'%'; Query OK, 0 rows affected (0.00 sec) |
Installing mysql_fdw on PostgreSQL server
Under the hood, MySQL FDW (mysql_fdw) facilitates the use of PostgreSQL server as a client for MySQL Server, which means it can then fetch data from the MySQL database as a client. Obviously, mysql_fdw uses MySQL client libraries. Nowadays, many Linux distributions are packaged with MariaDB® libraries. This works well enough for mysql_fdw to function. If we install mysql_fdw from the PGDG repo, then mariadb-devel.x86_64 packages will be installed alongside other development packages. To switch to Percona packages as client libraries, you need to have the Percona development packages too.
1 | sudo yum install Percona-Server-devel-57-5.7.22-22.1.el7.x86_64.rpm |
Now we should be able to install the mysql_fdw from PGDG repository:
1 | sudo yum install mysql_fdw_10.x86_64 |
Connect to the PostgreSQL server where we are going to create the foreign table, and using the command-line tool, create mysql_fdw extension:
1 2 | postgres=# create extension mysql_fdw; CREATE EXTENSION |
Create a server definition to point to the MySQL server running on a host machine by specifying the hostname and port:
1 2 | postgres=# CREATE SERVER mysql_svr FOREIGN DATA WRAPPER mysql_fdw OPTIONS (host 'hr',port '3306'); CREATE SERVER |
Now we can create a user mapping. This maps the database user in PostgreSQL to the user on the remote server (MySQL). While creating the user mapping, we need to specify the user credentials for the MySQL server as shown below. For this demonstration, we are using PUBLIC user in PostgreSQL. However, we could use a specific user as an alternative.
1 2 | postgres=# CREATE USER MAPPING FOR PUBLIC SERVER mysql_svr OPTIONS (username 'fdw_user',password 'Secret!123'); CREATE USER MAPPING |
Import schema objects
Once we complete the user mapping, we can import the foreign schema.
1 | postgres=# IMPORT FOREIGN SCHEMA hrdb FROM SERVER mysql_svr INTO public; |
Or we have the option to import only selected tables from the foreign schema.
1 | postgres=# IMPORT FOREIGN SCHEMA hrdb limit to ("EMP","DEPT") FROM SERVER mysql_svr INTO public; |
This statement says that the tables “EMP” and “DEPT” from the foreign schema named “hrdb” in mysql_server need to be imported into the public schema of the PostgreSQL database.
FDWs in PostgreSQL allows us to import the tables to any schema in postgres.
Let’s create a schema in postgres:
1 2 3 | postgres=# create schema hrdb; postgres=# IMPORT FOREIGN SCHEMA hrdb limit to ("EMP","DEPT") FROM SERVER mysql_svr INTO hrdb; |
Suppose we need the foreign table to be part of multiple schemas of PostgreSQL. Yes, it is possible.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | postgres=# create schema payroll; CREATE SCHEMA postgres=# create schema finance; CREATE SCHEMA postgres=# create schema sales; CREATE SCHEMA postgres=# IMPORT FOREIGN SCHEMA hrdb limit to ("EMP","DEPT") FROM SERVER mysql_svr INTO payroll; IMPORT FOREIGN SCHEMA postgres=# IMPORT FOREIGN SCHEMA hrdb limit to ("EMP","DEPT") FROM SERVER mysql_svr INTO finance; IMPORT FOREIGN SCHEMA postgres=# IMPORT FOREIGN SCHEMA hrdb limit to ("EMP","DEPT") FROM SERVER mysql_svr INTO sales; IMPORT FOREIGN SCHEMA |
You might be wondering if there’s a benefit to doing this. Yes, since in a multi-tenant environment, it allows us to centralize many of the master/lookup tables. These can even sit in a remote server, and that can be MySQL as well!.
IMPORTANT: PostgreSQL extensions are database specific. So if you have more than one database inside a PostgreSQL instance/cluster, you have to create a separate fdw extension, foreign server definition, and user mapping.
Foreign tables with a subset of columns
Another important property of foreign tables is that you can have a subset of columns if you are not planning to issue DMLs on the remote table. For example, MySQL’s famous sample database Sakila contains a table “film” with the following definition
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | CREATE TABLE `film` ( `film_id` smallint(5) unsigned NOT NULL AUTO_INCREMENT, `title` varchar(255) NOT NULL, `description` text, `release_year` year(4) DEFAULT NULL, `language_id` tinyint(3) unsigned NOT NULL, `original_language_id` tinyint(3) unsigned DEFAULT NULL, `rental_duration` tinyint(3) unsigned NOT NULL DEFAULT '3', `rental_rate` decimal(4,2) NOT NULL DEFAULT '4.99', `length` smallint(5) unsigned DEFAULT NULL, `replacement_cost` decimal(5,2) NOT NULL DEFAULT '19.99', `rating` enum('G','PG','PG-13','R','NC-17') DEFAULT 'G', `special_features` set('Trailers','Commentaries','Deleted Scenes','Behind the Scenes') DEFAULT NULL, `last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, PRIMARY KEY (`film_id`), KEY `idx_title` (`title`), KEY `idx_fk_language_id` (`language_id`), KEY `idx_fk_original_language_id` (`original_language_id`), CONSTRAINT `fk_film_language` FOREIGN KEY (`language_id`) REFERENCES `language` (`language_id`) ON UPDATE CASCADE, CONSTRAINT `fk_film_language_original` FOREIGN KEY (`original_language_id`) REFERENCES `language` (`language_id`) ON UPDATE CASCADE ) ENGINE=InnoDB AUTO_INCREMENT=1001 DEFAULT CHARSET=utf8 |
Imagine that we don’t need all of these fields to be available to the PostgreSQL database and its application. In such cases, we can create a foreign table with only the necessary columns in the PostgreSQL side. For example:
1 2 3 4 | CREATE FOREIGN TABLE film ( film_id smallint NOT NULL, title varchar(255) NOT NULL, ) SERVER mysql_svr OPTIONS (dbname 'sakila', table_name 'film'); |
The challenges of incompatible syntax and datatypes
There are many syntactical differences between MySQL and PostgreSQL. Consequently, you may need to manually intervene to create foreign tables. For example, MySQL tables accept the definition of enumerations in place, whereas PostgreSQL expects enumeration types to be defined before creating the table like this:
1 | CREATE TYPE rating_t AS enum('G','PG','PG-13','R','NC-17'); |
Many such things are not handled perfectly. So it is better to specify them as a text datatype. The same applies to the set datatype.
1 2 3 4 5 6 | CREATE FOREIGN TABLE film ( film_id smallint NOT NULL, title varchar(255) NOT NULL, rating text, special_features text ) SERVER mysql_svr OPTIONS (dbname 'sakila', table_name 'film'); |
I’m used to receiving skepticism from people about treating enum and set as text. Well, please don’t forget that we are not storing them in PostgreSQL, the text data type is just a method for handling input and output from the table. The data is pulled and pushed from the foreign server, which is MySQL, and this converts these texts into the corresponding enumeration before storing them.
IMPORTANT: mysql_fdw has the capability to do data type conversion (casting) automatically behind the scenes when a user fires DML against foreign tables.
Generally, DML against a remote MySQL database from the PostgreSQL side can be quite challenging because of the architecture differences. These impose restrictions, such as the first column of the foreign table must be unique. We will cover these in more depth in a future post.
Handling views on the MySQL side
While foreign tables are not limited tables on the MySQL side, a view can also be mapped as a foreign table. Let’s create a view in the MySQL database.
1 | mysql> create view v_film as select film_id,title,description,release_year from film; |
PostgreSQL can treat this view as a foreign table:
1 2 3 4 5 6 | postgres=# CREATE FOREIGN TABLE v_film ( film_id smallint, title varchar(255) NOT NULL, description text, release_year smallint ) SERVER mysql_svr OPTIONS (dbname 'sakila', table_name 'v_film'); CREATE FOREIGN TABLE |
Views on the top of foreign table on PostgreSQL
PostgreSQL allows us to create views on the top of foreign tables. This might even be pointing to a view on the remote MySQL server. Let’s try creating a view using the newly created foreign table v_film.
1 2 3 4 5 6 7 8 9 10 11 | postgres=# create view v2_film as select film_id,title from v_film; postgres=# explain verbose select * from v2_film; QUERY PLAN -------------------------------------------------------------------------- Foreign Scan on public.v_film (cost=10.00..1010.00 rows=1000 width=518) Output: v_film.film_id, v_film.title Local server startup cost: 10 Remote query: SELECT `film_id`, `title` FROM `sakila`.`v_film` (4 rows) |
Materializing the foreign tables (Materialized Views)
One of the key features mysql_fdw implements is the ability to support persistent connections. After query execution, the connection to the remote MySQL database is not dropped. Instead, it retains the connection for the next query from the same session. Nevertheless, in some situations, there will be concerns about continuously streaming data from the source database (MySQL) to the destination (PostgreSQL). If you have a frequent need to access data from foreign tables, you could consider the option of materializing the data locally. It is possible to create a materialized view on the top of the foreign table.
1 2 | postgres=# CREATE MATERIALIZED VIEW mv_film as select * from film; SELECT 1000 |
Whenever required, we can just refresh the materialized view.
1 2 | postgres=# REFRESH MATERIALIZED VIEW mv_film; REFRESH MATERIALIZED VIEW |
Automated Cleanup
One of the features I love about the FDW framework is its ability to clean up foreign tables in a single shot. This is very useful when we setup foreign table for a temporary purpose, like data migration. At the very top level, we can drop the extension, PostgreSQL will walk through the dependencies and drop those too.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | postgres=# drop extension mysql_fdw cascade; NOTICE: drop cascades to 12 other objects DETAIL: drop cascades to server mysql_svr drop cascades to user mapping for public on server mysql_svr drop cascades to foreign table "DEPT" drop cascades to foreign table "EMP" drop cascades to foreign table hrdb."DEPT" drop cascades to foreign table hrdb."EMP" drop cascades to foreign table payroll."DEPT" drop cascades to foreign table payroll."EMP" drop cascades to foreign table finance."DEPT" drop cascades to foreign table finance."EMP" drop cascades to foreign table sales."DEPT" drop cascades to foreign table sales."EMP" DROP EXTENSION postgres=# |
Conclusion
I should concede that the features offered by mysql_fdw are far fewer compared to postgres_fdw. Many of the features are not yet implemented, including column renaming. But the good news is that the key developer and maintainer of mysql_fdw is here with Percona! Hopefully, we will be able to put more effort into implementing some of the missing features. Even so, we can see here that the features implemented so far are powerful enough to support system integration. We can really make the two sing together!
Percona’s support for PostgreSQL
As part of our commitment to being unbiased champions of the open source database eco-system, Percona offers support for PostgreSQL – you can read more about that here.
You May Also Like
MySQL and PostgreSQL make a formidable tandem, but each open source database has its own strengths and weaknesses. Getting the most out of your databases requires an understanding of those traits. Our white papers on MySQL and PostgreSQL can give your business the insight it needs to maximize your data environment, helping you learn how to best utilize these open source databases. For more information, see these links:
nice! If the MYSQL schema changes, how are they propagated? what happens if there’s (new) impedance mismatches (e.g. incompatible syntax / datatypes).
Then you need to drop the PostgreSQL foreign table and create a new one with matched MySQL table. Your data in mysql will be intact by dropping the PostgreSQL foreign table.
1) how do you know you need to do this? can it be automated?
I prefer not to automate that, as we saw in the section “Foreign tables with a subset of columns” in the article, we may not need every data from remote database to be present in PostgreSQL side.
We can pull only minimal information which is required to satisfy our business needs. For example, A payroll system running in PostgreSQL can use essential data from HR system.
Another advantage is many of the changes happening in the source system (MySQL) may not directly impact PostgreSQL side, such as adding new column to table in MySQL which is not required at the PostgreSQL side.
2) what if you have non-trivial amounts of data? seems crazy to re-import everything?
3) if this is done transactionally in Postgres, are you holding table locks? How do you avoid deadlocks/timeouts?
i.e. how practical is all of this?
2) what if you have non-trivial amounts of data? seems crazy to re-import everything?
Data never gets fully “imported”/stored into PostgreSQL side, only table definitions gets imported. PostgreSQL serer acts just like a MySQL client. PostgreSQL pulls only essential data from MySQL side based on query/join optimization.
if query on remote table is very frequent and if you want to avoid streaming data as part of query every time, you can consider the materializing them at PostgreSQL side. Please refer the section “Materializing the foreign tables (Materialized Views)” for the same
3) if this is done transitionally in Postgres, are you holding table locks? How do you avoid deadlocks/timeouts?
As we discussed, PostgreSQL server connects to MySQL server as a client. So automatically it is transaction compliant.
All server side strategies including locking is applicable as PostgreSQL is acting a client. Completely avoiding deadlocks is not possible in any of the transaction systems, however we can avoid the chance by smart designs.
i.e. how practical is all of this?
FDWs are in general, practically used very much. It is key part in many system-integrations and migrations. It is even used as a simple sharding technique. However, It is not well known outside PostgreSQL circles, Just because SQL/MED feature are not present in my databases. Purpose of this blog is to make it more popular 🙂