When top shows high wa (I/O wait) or the load average climbs while the CPU is idle, the disk is the suspect. iostat shows how hard each disk is working, and, more importantly, how long each request takes.

Install

iostat is part of sysstat:

sudo apt install sysstat      # Debian/Ubuntu
sudo dnf install sysstat      # RHEL family

The command to remember

iostat -xz 1
  • -x — extended statistics (the useful columns)
  • -z — hide idle devices
  • 1 — refresh every second

As with vmstat, the first report shows averages since boot. Watch the following ones.

Device   r/s    w/s   rkB/s   wkB/s  rrqm/s wrqm/s r_await w_await aqu-sz  %util
vda     12.0  410.0   480.0 18200.0    0.0   95.0    0.85   14.20   5.83   97.6

The columns that matter

Column Meaning
r/s, w/s Read and write requests per second (IOPS)
rkB/s, wkB/s Read and write throughput
r_await, w_await Average time per read or write request, in ms, including time queued. This is the latency your apps feel
aqu-sz Average queue length. Sustained values well above 1 mean requests are piling up
%util Share of time the device had at least one request in flight
rrqm/s, wrqm/s Requests merged before being sent to the device

Reading latency

Rough guides for await:

  • SSD/NVMe: under ~1–2 ms is good. Tens of ms means the device is saturated or throttled.
  • HDD: 5–15 ms is normal. 50 ms or more means trouble.

Don’t over-trust %util

%util near 100% was a reliable saturation signal for single spinning disks. SSDs, NVMe drives and virtual disks handle many requests in parallel, so they can show 100% util while still having capacity. Judge by await and aqu-sz rising together with load. In the example above, 410 writes/s at 14 ms each with a queue of ~6 is a disk that can’t keep up with writes.

Find who is doing the I/O

iostat tells you which disk. To find which process:

sudo iotop -oPa          # only processes doing I/O, accumulated totals
pidstat -d 1             # per-process read/write kB/s (from sysstat)

Typical culprits on servers:

  • Databases with too little memory for their cache, so they read from disk constantly. Give the buffer pool more RAM. See MySQL buffer pool.
  • Swapping. Check si/so in vmstat.
  • Backups and rsync jobs during peak hours. Schedule them off-peak and run them with ionice -c3.
  • Logging storms: debug logging, or access logs under bot traffic.
  • Large file operations: tar, gzip, find over millions of files.

Throttle a heavy job

ionice -c3 nice -n19 tar -czf /backup/site.tar.gz /var/www/site     # idle I/O class, lowest CPU priority

The idle class (-c3) only gets disk time when nothing else wants it. The BFQ and CFQ schedulers honour it fully. Under the default mq-deadline and none schedulers on many virtual disks, it has limited effect, so scheduling heavy jobs off-peak is more reliable.

More iostat options

iostat -xz -p vda 1        # include partitions of vda
iostat -xzm 1              # throughput in MB/s
iostat -c 1                # CPU only
iostat -xz 5 12            # 12 samples, 5 s apart (one minute)
iostat -t -xz 1            # with timestamps

History with sar

sysstat can record statistics continuously, which is very useful for “what happened at 3am?”:

sudo systemctl enable --now sysstat
sar -d -p                       # today's disk stats (10-minute samples)
sar -d -p -f /var/log/sysstat/sa14     # the 14th of the month (Debian path)

On Debian/Ubuntu, also set ENABLED="true" in /etc/default/sysstat.

Quick benchmark (careful)

To see what a disk can do, use fio on a test file, never on a raw device that holds data:

fio --name=randrw --filename=/tmp/fio.test --size=1G --rw=randrw --bs=4k --iodepth=32 \
    --ioengine=libaio --direct=1 --runtime=30 --time_based --group_reporting
rm /tmp/fio.test

Benchmarks put load on shared storage. Keep them short.

Quick reference

Task Command
Live extended stats iostat -xz 1
Per process I/O sudo iotop -oPa / pidstat -d 1
Lower a job’s I/O priority ionice -c3 cmd
Historical sar -d -p
Latency to watch r_await, w_await
Saturation signs rising aqu-sz + await

Related: load average and Linux disk performance.

  • Linux
  • Performance
  • Storage