iostat -x

My favorite Linux tool in DB work is ‘iostat -x’ (and I really really want to see whenever I’m doing any kind of performance analysis), yet I had to learn its limitations and properties. For example, I took 1s snapshot from a slightly overloaded 16-disk database box:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           8.12    0.00    2.57   21.65    0.00   67.66

Device:  rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s \
sda     7684.00    19.00 2420.00  498.00 81848.00  5287.00 \

        avgrq-sz avgqu-sz   await  svctm  %util
           29.86    32.99   11.17   0.34 100.00

I pasted this somewhere on IRC, and got “doesn’t look too healthy” and that it is disk-bound. Now, to understand if it really is, one has to understand what iostat tells here.

First line of numbers shows that we’ve got plenty of CPU resources (thats because nowadays it is quite difficult to get a box with not enough CPU power, and I/O still seems to be bottleneck) – and we have more threads waiting for I/O than we have CPU execution (that sounds normal).

Now the actual per-disk statistics are where one should look. I used to prefer %util over general %iowait (I couldn’t really explain what %iostat is, and I can say what %util is). I don’t know why, but iostat has most interesting bits at the end, and not so interesting at the start:

  • %util: how much time did the storage device have outstanding work (was busy). In proper RAID environments it is more like “how much time did at least one disk in RAID array have something to do”. I’m deliberately excluding any kind of cache here – if request can be served from cache, the chance is quite negligible it will show up in %util, unlike in other values. What this also means – the RAID subsystem can be loaded from 6.25% (one disk doing the work) to 100% (all of them busy). Thats quite a lot of insight in single value of ‘100%’, isn’t it?
  • svctm: Though manual says “The average service time (in milliseconds) for I/O requests that were issued to the device.”, it isn’t exactly that when you look at multiple-disk systems. What it says is, “when your I/O subsystem is busy, how fast does it respond requests overall”. Actually, less you load your system, higher svctm is (as there’re less outstanding requests, and average time to serve them goes up). Of course, at some certain moment, when I/O becomes really overloaded, you can see svctm going up. One can tweak /sys/block/sda/queue/nr_requests based on this – to avoid overloading I/O controller, though that is really rarely needed.
  • await. One of my favorites – how fast do requests go through. It is just an average, how long it takes to serve a request for a device, once it gets into device queue, to final “OK”. Low = good, high = bad. There’re few gotchas here – even though different reads can have different performance properties (middle of disk, outer areas of disk, etc), the biggest difference is between reads and writes. Reads take time, writes can be instant (write caching at underlying layers..). As 80% of requests were reads, we can try to account for that by doing 11.17/0.8 math, to get 14ms figure. Thats quite high – systems that aren’t loaded can show ~5ms times (which isn’t that far away from 4ms rotation time of 15krpm disk).
  • avgqu-sz: Very very very important value – how many requests are there in a request queue. Low = either your system is not loaded, or has serialized I/O and cannot utilize underlying storage properly. High = your software stack is scalable enough to load properly underlying I/O. Queue size equal to amount of disks means (in best case of request distribution) that all your disks are busy. Queue size higher than amount of disks means that you are already trading I/O response time for better throughput (disks can optimize order of operations if they know them beforehand, thats what NCQ – Native Command Queueing does). If one complains about I/O performance issues when avgqu-sz is lower, then it is application specific stuff, that can be resolved with more aggressive read-ahead, less fsyncs, etc. One interesting part – avqu-sz, await, svctm and %util are iterdependent ( await = avgqu-sz * svctm / (%util/100)
  • avgrq-sz: Just an average request size. Quite often will look like a block size of some kind – can indicate what kind of workload happens. This is already post-merging, so lots of adjacent block operations will bump this up. Also, if database page is 16k, though filesystem or volume manager block is 32k, this will be seen in avgrq-sz. Large requests indicate there’s some big batch/stream task going on.
  • wsec/s & rsec/s: Sectors read and written per second. Divide by 2048, and you’ll get megabytes per second. I wanted to write this isn’t important, but remembered all the non-database people who store videos on filesystems :) So, if megabytes per second matter, these values are important (and can be seen in ‘vmstat’ output too). If not, for various database people there are other ones:
  • r/s & w/s: Read and write requests per second. This is already post-merging, and in proper I/O setups reads will mean blocking random read (serial reads are quite often merged), and writes will mean non-blocking random write (as underlying cache can allow to serve the OS instantly). These numbers are the ones that are the I/O capacity figures, though of course, depending on how much pressure underlying I/O subsystem gets (queue size!), they can vary. And as mentioned above, on rotational media it is possible to trade response time (which is not that important in parallel workloads) for better throughput.
  • rrqm/s & wrqm/s: How many requests were merged by block layer. In ideal world, there should be no merges at I/O level, because applications would have done it ages ago. Ideals differ though, for others it is good to have kernel doing this job, so they don’t have to do it inside application. Quite often there will be way less merges, because applications which tend to write adjacent blocks, also tend to wait after every write (see my rant on I/O schedulers). Reads however can be merged way easier – especially if application does “read ahead” block by block. Another reason for merges is simple block size mismatch – 16k database pages on top of 8k database pages will cause adjacent block reads, which would be merged by block layer. On some systems read of two adjacent pages would result in 1MB reads, but thats another rant :)
  • Device: – just to make sure, that you’re looking at the right device. :-)

So, after all this, the iostat output above tells us something like:

  • System has healthy high load (request queue has two-requests-per-disk)
  • Average request time is double the value one would expect from idle system, it isn’t too harmful, but one can do better
  • It is reading 80 40MB/s from disks, at 2420 requests/s. Thats quite high performance from inexpensive 2u database server (shameless plug: X4240 :)
  • High amount of merges comes from LVM snapshots, can be ignored
  • System is alive, healthy and kicking, no matter what anyone says :)
This entry was posted in mysql and tagged , , . Bookmark the permalink.

18 Responses to iostat -x

  1. Don McArthur says:

    For the information of readers using CentOS and unable to find iostat installed or via yum, do ‘yum install sysstat’ instead.

  2. Didier Spezia says:

    Actually, it is reading 40 MB/s (81848.00/2048), and not 80 MB/s.

  3. Matt Y says:

    Could agree more, I love, no I ABSOLUTLEY LOVE IT! Iostat its one of the most useful tools I have found:) Its truly amazing how many people have never seen it or used it before.

  4. David Cartwright says:

    @Don

    Thanks.

    yum install sysstat

    is also needed on Fedora 10.

  5. Mike MacCana says:

    From the article:
    “Actually, less you load your system, higher svctm is (as there’re more outstanding requests, and average time to serve them goes up). ”

    Are you sure about that? Why would less load result in more outstanding requests and higher service time?

  6. @Mike, ergh, svctm would go up because of _less_ outstanding parallel requests (more parallelism = higher overall throughput)

  7. Mike MacCana says:

    @Domas: I know that’s what you’re saying, but it seems to differ from the documentation. According to the man page service time is ‘The average service time (in milliseconds) for I/O requests that were issued to the device.’ I read this as the time taken to respond to requests (hence my statement that slower response = more service time), but this may be misleading – I gather you’re saying it’s the opposite, but why? Is the man page wrong or misleading?

  8. Mike MacCana says:

    Or, to make it short: where is it documented that svctime is a measure of throughput, and why does this differ from ‘time to be served’ per the manual?

  9. @Mike, I didn’t write the manual ;-) The formula for svctime is quite clear ( svctm = (await*%util)/avgqu-sz * 100) ) ) – so, bigger queue is, smaller svctm is.

  10. Stephen P. Schaefer says:

    Assuming (large assumption) that requests are for random places on the disk, I would expect the elevator algorithm to cause less wait for disk head motion per request for a larger number of requests than for a smaller number. The assumption may be entirely wrong; for all I know, there is naturally emerging locality of activity, and it will of course differ between database indexes and video delivery.

  11. Kord Campbell says:

    If you want to graph the results over time, download the free version of Splunk from their website and then add a small scripted input to run ‘iostat -x’ every minute or so.

    You’ll end up using a command called |multikv to split the lines and fields out for graphing. Usually takes about 15 minutes to get all of it running – and it’s FREE!

  12. I think you misspelt %iowait as %iostat in one place.

    Personally, I like %iowait. It’s the amount of time that your CPU(s) is (are) idling because all runnable programs are blocked waiting for I/O. It’s useful when you want to get an overview of what your system is doing, and perhaps less useful if you’re only interested in the I/O side.

  13. Mike MacCana says:

    hehe, I know you didn’t write the manual Domas, but it could be worth reporting the upstream documentation as being a little misleading.

  14. The device utilization (%util) doesn’t let you know anything about how the underlying physical devices are used (unless your logical device maps to exactly one physical device).

    What the device utilization shows is the percentage of time between two measurements, for which the logical device had outstanding I/Os (hence: “Was busy”). See Documentation/iostats.txt in Linux and the iostat source for details.

    Nice summary though, highly appreciated!

  15. Isotopp says:

    A relatively new tool in the Linux toolbox that I find increasingly useful is btrace (part of the sysstat package) on a sufficiently recent kernel.

  16. Pingback: Dev Blog AF83 » Blog Archive » Veille technologique : Design, Navigateurs, Optimisations, Javascript, Ruby, Rails, Agilité, Outils

  17. Pingback: Interesting Items - April 27, 2009 | Oxidiser techblog

Comments are closed.