KVM I/O slowness on RHEL 6

Written by Gionatan Danti on . Posted in Virtualization

User Rating:  / 105
PoorBest 

Over one year has passed since my last virtual machine hypervisor comparison so, in the last week, I was preparing an article showing a face to face comparison between RHEL 6 KVM technologies versus Oracle VirtualBox 4.0 product. I spent several days creating some nice, automated script to evaluate these two products under different point of views, and I was quite confident that the benchmark session would be completed without too much trouble. So, I installed Red Hat Enterprise Linux 6 (license courtesy of Red Hat Inc. - thank you guys!) on my workstation and I begin the virtual images installation.

However, the unexpected happened: using KVM, a Windows Server 2008 R2 Foundation installation took almost 3 hours, while normally it should be completed in about 30-45 minutes. Similarly, the installation of the base system anticipating the “real” Debian 6.0 installation took over 5 minutes, when normally it can be completed in about 1 minute. In short: the KVM virtual machines were affected by awfully slow disk I/O subsystem. In previous tests, I saw that KVM I/O subsystem was a bit slower, but not by so much; clearly, something was impairing my KVM I/O speed. I tried different combination of virtualized disk controllers (IDE or VirtIO) and cache settings, but without success. I also changed my physical disk filesystem to EXT3, to avoid any possible, hypothetical EXT4 speed regression, but again with no results: the KVM slow I/O speed problem remained.

I needed a solution – and a real one: with such awfully slow I/O, the KVM guests were virtually unusable. After some wasted hours, I decided to run some targeted, systematic tests regarding VM image formats, disk controllers, cache settings and preallocation policy. Now that I find the solution and I run my KVM guest at full speed, I am very happy – and I would like to share my results with you.

Comments   

 
#1 lawrence 2012-04-27 00:05
very helpful.

thanks.
 
 
#2 Josh 2012-05-10 15:37
This is a GREAT article. However... splitting it up into 8 pages makes it more annoying to use. I would understand doing this, if you had advertising on each page and wanted to derive advertising revenue. But as it stands I've copied all 8 pages into an OpenOffice document so I can turn it into a PDF and file it with the rest of my "useful blog posts".
 
 
#3 Gionatan Danti 2012-05-10 17:01
Quoting Josh:
This is a GREAT article. However... splitting it up into 8 pages makes it more annoying to use. I would understand doing this, if you had advertising on each page and wanted to derive advertising revenue. But as it stands I've copied all 8 pages into an OpenOffice document so I can turn it into a PDF and file it with the rest of my "useful blog posts".


Hi Josh, you are right: while a single page view would be desiderable, the only manner to cover domain's costs is through advertising.

Feel free to store a your personal copy of the article in whatever format you want.

Regads.
 
 
#4 Morrizor 2012-05-16 18:46
Thanks Gionatan, very useful!
 
 
#5 Arturs 2012-06-03 20:36
Hy, it very helpful!
I tried myself with default server is centos 5.8 x86_64 is installing for long long time, formating image is very long!

But when i tried cache none on full(full size already allocated) image, its running perfect
 
 
#6 Siauhoo 2012-10-06 09:58
It seem that raw images doesn't support preallocation. if so why there is raw + preallocation in this chart ?
 
 
#7 hakl 2013-12-08 14:51
And where was the big tip which you told on first page?
 
 
#8 Gionatan Danti 2013-12-16 17:56
Quoting hakl:
And where was the big tip which you told on first page?


Hi, the tip was to preallocate the QCOW2 backing file (as explained in the article) and to _not_ use the writethrough cache setting.

However, newer KVM/Qemu version (starting with those found in CentOS 6.1) greatly improved the usability of non-preallocate d QCOW2 files.

So, if you take the route to go with QCOW2 and you need thin provision, you can avoid to preallocate the backing file. However, remember to avoid the writethough cache.

Regards.
 

You have no rights to post comments