In a previous blog post: “Tuning Linux for MongoDB,” I covered several tunings for an efficient MongoDB deployment on Linux in Production. This post expands on that one.
While I felt the tuning Linux for MongoDB was a very useful blog post that results in a great baseline tuning, something bugged me about how much effort and touch-points were required to achieve an efficient Linux installation for MongoDB. More importantly, I noticed some cases where the tunings (example: changes to disk I/O scheduler in /etc/udev.d) were ignored on some recent RedHat and CentOS versions. With these issues in mind, I started to investigate better solutions for achieving the tuned baseline.
Tuned
In RedHat (and thus CentOS) 7.0, a daemon called “tuned” was introduced as a unified system for applying tunings to Linux. tuned operates with simple, file-based tuning “profiles” and provides an admin command-line interface named “tuned-adm” for applying, listing and even recommending tuned profiles.
Some operational benefits of tuned:
- File-based configuration – Profile tunings are contained in a simple, consolidated files
- Swappable profiles – Profiles are easily changed back/forth
- Standards compliance – Using tuned profiles ensures tunings are not overridden or ignored
Note: If you use configuration management systems like Puppet, Chef, Salt, Ansible, etc., I suggest you configure those systems to deploy tunings via tuned profiles instead of applying tunings directly, as tuned will likely start to fight this automation, overriding the changes.
The default available tuned profiles (as of RedHat 7.2.1511) are:
- balanced
- desktop
- latency-performance
- network-latency
- network-throughput
- powersave
- throughput-performance
- virtual-guest
- virtual-host
The profiles that are generally interesting for database usage are:
- latency-performance
“A server profile for typical latency performance tuning. This profile disables dynamic tuning mechanisms and transparent hugepages. It uses the performance governer for p-states through cpuspeed, and sets the I/O scheduler to deadline.”
- throughput-performance
“A server profile for typical throughput performance tuning. It disables tuned and ktune power saving mechanisms, enables sysctl settings that improve the throughput performance of your disk and network I/O, and switches to the deadline scheduler. CPU governor is set to performance.”
- network-latency – Includes “latency-performance,” disables transparent_hugepages, disables NUMA balancing and enables some latency-based network tunings.
- network-throughput – Includes “throughput-performance” and increases network stack buffer sizes.
I find “network-latency” is the closest match to our recommended tunings, but some additional changes are still required.
The good news is tuned was designed to be flexible, so I decided to make a MongoDB-specific profile: enter “tuned-percona-mongodb”.
tuned-percona-mongodb
tuned-percona-mongodb: https://github.com/Percona-Lab/tuned-percona-mongodb
“tuned-percona-mongodb” is a performance-focused tuned profile for MongoDB on Linux, and is currently considered experimental (no gurantees/warranties). It’s hosted in our Percona-Lab Github repo.
tuned-percona-mongodb applies the following tunings (from the previous tuning article) on a Redhat/CentOS 7+ host:
- Disabling of transparent huge pages
- Kernel network tunings (sysctls)
- Virtual memory dirty ratio changes (sysctls)
- Virtual memory “swappiness” (sysctls)
- Block-device readahead settings (on all disks except /dev/sda by default)
- Block-device I/O scheduler (on all disks except /dev/sda by default)
The following tunings that our previous tuning article didn’t cover are also applied:
- CPU C-States and Energy-Bias tunings for high performance
- Disk Aggressive Link Power Management (ALPM) tuning, if available/supported
After a successful deployment of this profile, only these recommendations are outstanding:
- Filesystem type and mount options:
Tuned does not handle filesystem mount options, this needs to be done manually in /etc/fstab. To quickly summarize: we recommend the XFS or EXT4 filesystem type for MongoDB data when using MMAPv1 or RocksDB storage engines, and XFS ONLY when using WiredTiger. For all filesystems, using the mount options “rw,noatime” will reduce some activity. - NUMA disabling or interleaving:
Tuned does not handle NUMA settings and these still need to be handled via the MongoDB init script or the BIOS on/off switch. - Linux ulimits:
Tuned does not set Linux ulimit settings. However, Percona Server for MongoDB RPM packages do this for you at startup! See “LimitNOFILE” and “LimitNPROC” in “/usr/lib/systemd/system/mongod.service” for more information. NTP server:
Tuned does not handle installation of RPM packages or enabling of services. You will need to install the “ntp” package and enable/start the “ntpd” service manually:123sudo yum install ntpsudo systemctl enable ntpdsudo systemctl start ntpd
tuned-percona-mongodb: Installation
The installation of this profile is as simple as checking-out the repository with a “git” command and then running “sudo make enable”, full output here:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | $ git clone https://github.com/Percona-Lab/tuned-percona-mongodb $ cd tuned-percona-mongodb $ sudo make enable if [ -d /etc/tuned ]; then cp -dpR percona-mongodb /etc/tuned/percona-mongodb; echo "### 'tuned-percona-mongodb' is installed. Enable with 'make enable'."; else echo "### ERROR: cannot find tuned config dir at /etc/tuned!"; exit 1; fi ### 'tuned-percona-mongodb' is installed. Enable with 'make enable'. tuned-adm profile percona-mongodb tuned-adm active Current active profile: percona-mongodb |
In the example above you can see “percona-mongodb” is now the active tuned profile on the system (mentioned on the last output line).
The tuned profile files are installed to “/etc/tuned/percona-mongodb”, as seen here:
1 2 3 | $ ls -alh /etc/tuned/percona-mongodb/*.* -rwxrwxr-x. 1 root root 677 Nov 22 20:00 percona-mongodb.sh -rw-rw-r--. 1 root root 1.4K Nov 22 20:00 tuned.conf |
Let’s check that the “deadline” i/o scheduler is now the current scheduler on any disk that isn’t /dev/sda (“sdb” used below):
1 2 | $ cat /sys/block/sdb/queue/scheduler noop [deadline] cfq |
Transparent huge pages should be disabled (it is!):
1 2 3 4 | $ cat /sys/kernel/mm/transparent_hugepage/enabled always madvise [never] $ cat /sys/kernel/mm/transparent_hugepage/defrag always madvise [never] |
Block-device readahead should be 32 (16kb) on /dev/sdb (looks good!):
1 2 | $ blockdev --getra /dev/sdb 32 |
That was easy!
tuned-percona-mongodb: Uninstallation
To uninstall the profile, run “sudo make uninstall” in the github checkout directory:
1 2 3 4 5 6 7 8 9 10 11 12 | if [ -d /etc/tuned/percona-mongodb ]; then echo "### Disabling tuned profile 'tuned-percona-mongodb'"; echo "### Changing tuned profile to 'latency-performance', adjust if necessary after!"; tuned-adm profile latency-performance; tuned-adm active; else echo "tuned-percona-mongodb profile not installed!"; fi ### Disabling tuned profile 'tuned-percona-mongodb' ### Changing tuned profile to 'latency-performance', adjust if necessary after! Current active profile: latency-performance if [ -d /etc/tuned/percona-mongodb ]; then rm -rf /etc/tuned/percona-mongodb; fi |
Note: the uninstallation will enable the “latency-performance” tuned profile, change this after the uninstall if needed
To confirm the uninstallation, let’s check if the block-device readahead is set back to default (256/128kb):
1 2 | $ sudo blockdev --getra /dev/sdb 256 |
Uninstall complete.
Conclusion
So far tuned shows a lot of promise for tuning Linux for MongoDB, providing a single, consistent interface for tuning the Linux operating system. In the future, I would like to see the documentation for tuned improve. However, its simplicity makes the need for documentation rarely necessary.
As mentioned, after applying “tuned-percona-mongodb” you still need to configure an NTP server, NUMA (in some cases) and the filesystem type+tunings manually. The majority of the time, effort and room for mistakes is greatly reduced using this method.
If you have any issues with this profile for tuning Linux for MongoDB, or have any questions, please create a Github issue at this URL: https://github.com/Percona-Lab/tuned-percona-mongodb/issues/new.
Nice article on Tim .
Is tuned profiles are available in Amazon linux. if not how can this be implemented on Amzon linux .
Hi Tim,
will you please tell us this “tuned-percona-mongodb” for MongoDB can be implemented on the amazon EC2 servers with amazon linux operating system
Thanks in advance