KVM I/O slowness on RHEL 6

Written by Gionatan Danti on . Posted in Virtualization

User Rating:  / 106
PoorBest 

On caching and preallocation

Note: in this page, I try to condensate in very little space some hard-to-explain concepts, so I had to do some approximations. I ask the expert reader to forgive me for the over-simplification.

Normally, a virtual guest system use an host-side file to store its data: this file represent a virtual disk, that the guest use as a normal, physical disk. However, from the host view this virtual disk is a normal data file and it may be subject to caching and preallocation.

In this context, caching is the process to “hide” some disk-related data to physical RAM. When we use that cache to storein RAM only data previously read from the disk, we speak about a read cache, or write-through cache. When we store in RAM some data that will be later flushed to disk, we speak abut a write cache, or write-back cache. A write-back cache, by caching write request in the fast RAM, has higher performance; however, it is also more prone to data loss than a write-through one, as the latter only cache read requests and immediately write to disk any data.

As disk I/O is a very important parameter, Linux and Windows o.s. generally use a write-back policy with periodic flush to the physical disk. However, when using an hypervisor to virtualize some guest system, you can effectively cache things twice (one time in the host memory and another time in the virtual guest memory), so is often better to disable host-based caching on the virtual disk file and to let the guest system to manage its own caching. Moreover, a host-side write-back policy on virtual disk file significantly increase the risk of data loss in case of guest crash.

KVM let you choose one of these three cache policy: no caching, write-through (read only cache) and write-back (read and write cache). It also has a “default” setting that effectively is an alias for the write-through one. As you will see, pick the right caching scheme is a crucial choice for fast guest I/O.

Now, some words about preallocation: this is the process to better prepare the virtual disk file to store the data written by the guest system. Generally, preallocate a file means to fill it with zeros, so that the host system had to reserve in advance all the disk space assigned to the guest. In this manner, when the guest try to write to the virtual disk, it never waits for the host system to reserve the required space. Some time, preallocation does not fill the target file with zeros, but only prepare some its internal data structure: in this case, we talk about metadata preallocation. RAW disk format can use full preallocation, while QCOW2 actually use metadata preallocation (there are some patches that force full preallocation, but are experimental ones).

Why speak about caching and preallocation? Because the super-slow KVM I/O speed really boil down to this two parameters, as we are going to see.

Comments   

 
#1 lawrence 2012-04-27 00:05
very helpful.

thanks.
 
 
#2 Josh 2012-05-10 15:37
This is a GREAT article. However... splitting it up into 8 pages makes it more annoying to use. I would understand doing this, if you had advertising on each page and wanted to derive advertising revenue. But as it stands I've copied all 8 pages into an OpenOffice document so I can turn it into a PDF and file it with the rest of my "useful blog posts".
 
 
#3 Gionatan Danti 2012-05-10 17:01
Quoting Josh:
This is a GREAT article. However... splitting it up into 8 pages makes it more annoying to use. I would understand doing this, if you had advertising on each page and wanted to derive advertising revenue. But as it stands I've copied all 8 pages into an OpenOffice document so I can turn it into a PDF and file it with the rest of my "useful blog posts".


Hi Josh, you are right: while a single page view would be desiderable, the only manner to cover domain's costs is through advertising.

Feel free to store a your personal copy of the article in whatever format you want.

Regads.
 
 
#4 Morrizor 2012-05-16 18:46
Thanks Gionatan, very useful!
 
 
#5 Arturs 2012-06-03 20:36
Hy, it very helpful!
I tried myself with default server is centos 5.8 x86_64 is installing for long long time, formating image is very long!

But when i tried cache none on full(full size already allocated) image, its running perfect
 
 
#6 Siauhoo 2012-10-06 09:58
It seem that raw images doesn't support preallocation. if so why there is raw + preallocation in this chart ?
 
 
#7 hakl 2013-12-08 14:51
And where was the big tip which you told on first page?
 
 
#8 Gionatan Danti 2013-12-16 17:56
Quoting hakl:
And where was the big tip which you told on first page?


Hi, the tip was to preallocate the QCOW2 backing file (as explained in the article) and to _not_ use the writethrough cache setting.

However, newer KVM/Qemu version (starting with those found in CentOS 6.1) greatly improved the usability of non-preallocate d QCOW2 files.

So, if you take the route to go with QCOW2 and you need thin provision, you can avoid to preallocate the backing file. However, remember to avoid the writethough cache.

Regards.
 

You have no rights to post comments