ZFS, BTRFS, XFS, EXT4 and LVM with KVM – a storage performance comparison

Testbed and methods

Benchmarks were performed on a system equipped with:
- PhenomII 940 CPU (4 cores @ 3.0 GHz, 1.8 GHz Northbridge and 6 MB L3 cache)
- 8 GB DDR2-800 DRAM (in unganged mode)
- Asus M4A78 Pro motherboard (AMD 780G + SB700 chipset)
- 4x 500 GB, 7200 RPM hard disks (1x WD RE, 3x Seagate Barracuda) in AHCI mode, configured in software RAID10 "near" layout (default 512K chunks)
- S.O. CentOS 7.0 x64, kernel version 3.10.0-123.13.2.el7.x86_64

All disk had 3 partitions, used for construct three MD arrays: a first RAID1 “boot” array, a RAID10 “system” one (/ + swap, via LVM) and a final “data” RAID 10 (~ 800 GB usable) array for VMs hosting and testing. BTRFS and ZFS natively support mirroring + striping, so in these cases I go ahead without the “data” MD array and used the integrated facilities to create a mirror+striped dataset.

ZFS was used with the kernel-level driver provided by ZFS on Linux (ZoL) project, version 0.6.3-1.2. A slight change in default setting was needed for 100% reliable operation: I had to enable the “xattr=sa” option. You can read more here: https://github.com/zfsonlinux/zfs/issues/1548

A benchmark run consisted in:
1) concurrently install the 4 VMs (using PXE on the host side + kickstart for unattended installation). The virtual machines had default configuration values, using LVM+XFS for guest systems. The four VM images were 8 GB each in size;
2) prepare a PostgreSQL database on one VM, using sysbench on the host side (using default parameters);
3) concurrently runs a different benchmark on each VM for ten minutes (600 seconds), three time in a row; the results were averaged.

Point n.3 need some more explanations:
- the first machine runs PostgreSQL and was benchmarked by a sysbench instance running on the host. I used the “complex” test with default options. The “complex” test is a mixed read/write, transactional test;
- the second machine runs filebench with fileserver personality, to simulate fileserver-type activity. It is a mixed read/write, mixed sequential/random test;
- the third machine runs filebench with varmail personality, issuing many small synchronized writes as many mailserver do;
- the fourth machine runs filebench with webserver personality, issuing many small reads as typically webservers do.

You can consider any synthetic or semi-synthetic test as flawed, and in some manner you are right. At the same time, the above tests are both easy reproducible (I used default options, both for configurations and benchmarks) and I/O heavy. Moreover, they represent and use a more-or-less credible I/O pattern.

You may wonder why I installed the 4 virtual machines concurrently, and why I run these very stressful tests all at the same time. The fact is that thin provision has a hidden threat: when issuing many I/O writes concurrently, data can be scattered all over the disk, leading to considerable fragmentation. Instead, when using a single VM at a time this behavior is usually not noticeably, as the system has no disk bandwidth contention in this case. The only exception was the database creation/population: it is a very specific operation that need to be issued to a single VM and in real world you will rarely load an entire DB during periods of high I/O usage.

One last thing: I did not specifically optimize guests configuration for maximum throughput. For example, the PostgreSQL service runs with default value for buffer size. For this benchmark round I was interested in creating a high I/O load, nor to fine-tune guests configuration.

• KVM Windows virtual machines and ACPI unattended shutdown: a recap
• Dell DSU and previous OMSA versions
• LVM thin volume explained
• Joomla 3.3, CentOS 6 and PHP version - putting all together
• Linux compressors comparison on CentOS 6.5 x86-64: lzo vs lz4 vs gzip vs bzip2 vs lzma