Vmware vs Virtualbox vs KVM vs XEN: virtual machines performance comparison

Written by Gionatan Danti on . Posted in Virtualization

User Rating:  / 167
PoorBest 

I/O benchmarks: IOMeter

The last synthetic test is IOMeter. It is a very interesting test because it can not only vary the chunk size (like HD Tune), but it can also vary the I/O queue depth (the number of outstanding requests per worker). Normally, varying the queue depth is used to test the NCQ capability of the chipset / disk combo; this time, we use it to test the various hypervisors. The block size is fixed to 4 KB.

Let see the IOPS number first:

IOMeter IOPS

The first half of the above graph is the read test, the second half is the write test.

The very first thing to note is that KVM is, by much, the slower hypervisor in the read test; VMware is at the other end of the spectrum with a very great performance. What can be the cause of that KVM poor show? It is not possible that the problem depend on the dynamic allocation of QCOW2 image, because IOMeter allocate all the needed disk space before the test execution. To me, the KVM results are due to very poor caching and great I/O overhead. Xen is considerably faster then KVM, but it lags behind VirtualBox by a great margin which is, in turn, at about half route between Xen and VMware. However, it is interesting to note that Xen and VMware are the only hypervisor that seem to have a little benefit from the increasing queue depth.

Watching the write test, we have a very different picture: the faster virtual machine is VirtualBox (with decreasing performance as queue depth rises), while the other seems to be equally slow. Don't be fooled by the graph however: there are very interesting difference between the other virtualizers, only the graph's scale is not fine enough. For give you a more precise view, I reported the raw write test data in a table:

 

1

2

4

8

16

32

XEN

410,28

367,6

370,37

374,77

376,66

376,78

KVM

108,52

107,22

105,81

109,3

107,17

95,8

VBOX

2022,38

8548,59

4756,33

3162,37

1120,3

2044,4

VMWARE

216,63

267,01

332,63

361,53

359,8

393,46

As you can see, KVM, Xen and VMWare behave in very different manners, but they are all much behind VirtualBox. Why? Remember the Windows 2008 installation time? The whole point can be a different write cacheing policy. To me, seems that VirtualBox use a write-back cache algorithm, while the others use a write-throught policy. The net results is greater speed for VirtualBox, but also a greater risk of data loss in the cases of power failure and/or guest/host crash. However, if you prefer speed over safety, the other virtualizers can be configured to use a write-back policy also.

Is worth note that, in write test, VMware is the only virtual machine that has a good boost from increased queue depth.

What about CPU load under Iometer tests?

IOMeter CPU load

As before, the left half is about the read test, the right half about the write test.

That graph shows that in the read test VirtualBox and VMware are the most CPU hungry hypervisor – but they have much greater I/O results than the others. In the write test, VirtualBox and KVM have the most CPU cost – but the perform in very different manner.

To give you a more correct point of view, I created a graph that visualizes the CPU load cost for each I/O operation:

IOMeter normalized CPU load

We can see a clear patter now: VirtualBox and VMware are the most I/O efficient virtualizers, with Xen not too much behind. KVM is the clear loser here.

UPDATE: a recent article comparing KVM vs VirtualBox can be found here: http://www.ilsistemista.net/index.php/virtualization/12-kvm-vs-virtualbox-40-on-rhel-6.html

Comments   

 
#1 Nathan 2012-09-12 03:12
This is a terrible review, to install the VMware paravirtual drivers but not the KVM Windows paravirtual drivers. All results from VMware must be discarded for comparison purposes.
 
 
#2 Marcelo 2015-11-15 03:16
A quick comparison I made between VMware Workstation Player and VirtualBox, with XP as guest, shows a ridiculous I/O advantage of VB, while VMware has a big advantage on 3D graphics.
 
 
#3 Gionatan Danti 2015-11-15 09:32
Quoting Marcelo:
A quick comparison I made between VMware Workstation Player and VirtualBox, with XP as guest, shows a ridiculous I/O advantage of VB, while VMware has a big advantage on 3D graphics.


Hi Marcelo,
VBox higher I/O speed probably is an artifact of VBox not honoring write barrier (synchronized writes) by default. While this give much higher speed, storage consistency is somewhat reduced and I do not suggest to disable write barriers on production host/machines.
 

You have no rights to post comments