Many types of I/O requests...
Before to have a look at the raw numbers, we must spent some minutes in characterizing the most common types of I/O requests that a system had to satisfy.
Read requests are often synchronous, blocking operations: if an application want to read a disk block, chances are that it will wait until the data are returned (unless it specifically requests an async I/O request, but it is a relatively rare case). This means that a good I/O scheduler will generally try to prioritize read traffic.
Write requests often are, on the other hand, asynchronous, non blocking operations: if an application want to write a disk block, the write will be generally cached inside an OS-mantained memory buffer located inside main system RAM, and it will be sent to disk on a later time.
However, not all writes behave identically: writes to files opened with O_DIRECT flag (for direct I/O) and O_SYNC (for synchronous I/O) will bypass the system cache. Moreover, some system-wide operation, as fsync(), can force a cache flush and so all writes will be immediately sent to disk.
How I/O scheduler performance can be affected by these different kind of I/O operations?
Standard writes will be cached by the OS buffers as long as possible, and then they will be sent to disk in few large batches. With these kind of writes the scheduler impact is going to be low, as it will be the batches size/position to mainly determine I/O performance. On the other hand, mixing read and write requests will impose additional strain on the scheduler, potentially exposing some differences.
Direct I/O requests will completely bypass the system cache, both for reads and writes, but they will hit the on-disk DRAM cache. In other words direct I/O writes, being cached by the internal disk cache, can return immediately. However, on-board disk caches are often limited in size and so it is crucial that the I/O scheduler do not pollute them with unneeded and/or deferrable requests. So, direct I/O is a good place to see how the schedulers perform.
Synchronous read/writes are, well, synchronous operations. This means that the I/O scheduler has little to no opportunity to optimize the I/O traffic. In addition to these different kind of read/write operations, a good scheduler has to evaluate request size also – prioritizing a 100GB read over a single 4KB one will means that the application asking for these few KBs of data will had to wait for a long, long time...