On caching and preallocation
Note: in this page, I try to condensate in very little space some hard-to-explain concepts, so I had to do some approximations. I ask the expert reader to forgive me for the over-simplification.
Normally, a virtual guest system use an host-side file to store its data: this file represent a virtual disk, that the guest use as a normal, physical disk. However, from the host view this virtual disk is a normal data file and it may be subject to caching and preallocation.
In this context, caching is the process to “hide” some disk-related data to physical RAM. When we use that cache to storein RAM only data previously read from the disk, we speak about a read cache, or write-through cache. When we store in RAM some data that will be later flushed to disk, we speak abut a write cache, or write-back cache. A write-back cache, by caching write request in the fast RAM, has higher performance; however, it is also more prone to data loss than a write-through one, as the latter only cache read requests and immediately write to disk any data.
As disk I/O is a very important parameter, Linux and Windows o.s. generally use a write-back policy with periodic flush to the physical disk. However, when using an hypervisor to virtualize some guest system, you can effectively cache things twice (one time in the host memory and another time in the virtual guest memory), so is often better to disable host-based caching on the virtual disk file and to let the guest system to manage its own caching. Moreover, a host-side write-back policy on virtual disk file significantly increase the risk of data loss in case of guest crash.
KVM let you choose one of these three cache policy: no caching, write-through (read only cache) and write-back (read and write cache). It also has a “default” setting that effectively is an alias for the write-through one. As you will see, pick the right caching scheme is a crucial choice for fast guest I/O.
Now, some words about preallocation: this is the process to better prepare the virtual disk file to store the data written by the guest system. Generally, preallocate a file means to fill it with zeros, so that the host system had to reserve in advance all the disk space assigned to the guest. In this manner, when the guest try to write to the virtual disk, it never waits for the host system to reserve the required space. Some time, preallocation does not fill the target file with zeros, but only prepare some its internal data structure: in this case, we talk about metadata preallocation. RAW disk format can use full preallocation, while QCOW2 actually use metadata preallocation (there are some patches that force full preallocation, but are experimental ones).
Why speak about caching and preallocation? Because the super-slow KVM I/O speed really boil down to this two parameters, as we are going to see.