Apparently 2.6.32 kernel, which is in Ubuntu 10.04 LTS (Lucid Lynx) and is also basis for future RHEL6 kernel has nearly 20% performance degradation for PG (though not for MySQL, phew) – and apparently it was news to their community (I have started an email thread, will see where it goes).

While I was doing my research, of course I could observe plenty of gems of wisdom:

<davidfetter> domas, well, as tom lane once said, there’s no limit to how quickly you can get an answer if it doesn’t have to be correct

And an hour later:

<rawtaz> are you one of those mysql lovers?
…
<dim> nobody wants performance at the price of correctness, right?
<dim> as Tom said, I can be as quick as you want as providing an answer if you don’t want it to be correct (hint: 42.)

Apparently chasing these performance variations is…

<Snow-Man> tbh, I’m of the opinion that we’re losing the forest for the trees.

Can’t say that all of them were encouraging:

<sterncera> If you can’t be bothered to subscribe to -hackers, maybe you shouldn’t be mailing it

Special thanks goes to mastermind, who not only didn’t lose his temper, but stayed focused on the topic and resisted my trolling :-) I really want PG to be greatly performing database (and I’ve seen some great numbers from it), but the amazing amount of ignorance and animosity they have against MySQL makes it somewhat complicated there though :(

P.S. Now all benchmarks I did are tainted and will need full rerun…

The synonym for ‘i/o scheduler’ inside Linux kernel is ‘elevator’ – and it helps to explain many things. The physical movement needs of disk spindles are similar to how elevators move people in tall buildings:

• Attempting of going single direction, just ‘up’ or ‘down’, and grabbing all people on their way.
• At every floor where it stops, waiting for more people to show up, not closing doors immediately.
• More full the elevator cabin is, more efficient it is in terms of transported people.
• More full the elevator cabin is, more annoying is for people in it – stopping at every floor, then waiting, people start hitting ‘close door’ button nervously.
• Buildings solve this by having more elevators, or sophisticated queueing systems for getting into them

Now, imagine a huge hotel building, that is having huge convention of privacy-worshippers. Or just misanthropes. They will never get into elevator, until they know that it is empty, and human that was traveling before got out of elevator safely. Essentially, thats how database transaction serialization works.

Thats where smart elevators fail – they immediately notice, that all writes are going to same location and prefer to wait for more requests – and merge them. Though, whenever an elevator decides to wait, nothing happens – there is a global lock inside database engine, which tells not to write until first write finishes.
Scheduler waits, decides that it did wait too long, flushes the write, gets another request, notices write goes to same location and there might be chance to merge subsequent requests, which… do not come in again.

And the solution is – using teleports. Well, at least treating database writes as instant accesses, not caring about order, waiting, just doing everything as soon as possible.

To demonstrate this I went to the world of edge cases – made the performance test, which maybe shouldn’t be called a ‘benchmark’. I created a very simple table and started spamming rows, each as a separate transaction, into it. The hardware for test was a ‘regular’ DB box, 8 disks, write-behind cache on RAID controller, 16GB of memory, 8 cores – not my desktop or laptop. The sole idea of test was finding how different I/O modes affect ability to write to I/O controller as fast as possible – resulting in transaction throughput.

Here are some results:

Here, CFQ had very huge regression at higher concurrency, actually Anticipatory showed similar, slightly better results. NOOP showed similar results to deadline, much faster at single thread, slightly slower at multiple.

So, whichever decisions CFQ takes during this test, they must be all wrong – with multiple disks and raid controller handling flushing of write cache there is no need for elevation or request merging, multiple tagged commands can be sent to the I/O subsystem, and they will be executed swiftly. It is supposed to be a scheduler good for most of workloads, and it probably is. But high-performance databases rely on storage being fast, and for enforcing of ACID requirements, synchronous operations should not wait forever (or wait, when not necessary)

Of course, CFQ may provide better performance on systems that do extensive I/O scanning, so for folks with slow queries it may end up providing more throughput, as it will tolerate delaying small things for big things to get through – deadline would not care about fairness and try getting everything done as quickly as possible.

The worst part is that Deadline is enabled by default just on community distributions, like Ubuntu Server (though Ubuntu desktop kernel has CFQ by default). So, most of people will end up having anti-database scheduler for ages, and will rarely get into internals of whole stack, or analysis of performance profile. Switching to another scheduler is a matter of single command (though it managed to crash my system once :)

Must read: Jens Axboe on his block layer work, and my previous rant :)