The server feels slow, the load average is high, but the CPU is mostly idle. top shows a big wa value. That’s I/O wait: processes are waiting for the disk. The CPU has nothing to do because the storage can’t keep up. Here’s how to find out who is hammering the disk and how to stop it.

What iowait means

wa in top is the percentage of time the CPU sat idle while at least one process waited for I/O. It’s a symptom, not a resource. A high wa means some work is blocked on disk. Processes in this state (D) also count towards the load average, which is why load climbs while the CPU looks idle. See load average.

Step 1: confirm it

top                 # %Cpu(s) … wa
vmstat 1 5          # 'b' = processes blocked on I/O, 'wa' column, 'bi/bo' = blocks in/out
 r  b   swpd   free  …   si   so    bi    bo …  us sy id wa st
 1  6 812000  42000  …  350  900  5200  3100 …   8  4 30 58  0

Here 6 processes are blocked and wa is 58%. Note the si/so values. This server is swapping, which is a very common cause.

Step 2: which disk, how bad?

iostat -xz 1

Watch r_await/w_await (latency in ms) and aqu-sz (queue). On SSD storage, awaits in the tens or hundreds of ms mean saturation. See iostat.

Step 3: which process?

sudo iotop -oPa          # only processes doing I/O, accumulated
pidstat -d 1             # per-process kB read/written per second
ps -eo state,pid,cmd | awk '$1 ~ /D/'     # processes stuck in I/O

Common causes and fixes

Swapping (the most common one)

If si/so are busy, the “disk problem” is really a memory problem. Pages are being moved between RAM and disk constantly. Reduce memory use (PHP-FPM children, database buffers, leaking apps) or add RAM. See high memory usage and swap.

Database reading from disk

When the working set doesn’t fit in the database’s cache, every query reads from disk:

  • MySQL: is innodb_buffer_pool_size too small for the hot data? Check the buffer pool hit rate. See buffer pool.
  • Missing indexes cause full table scans that read entire tables. See slow query log and indexing.
  • PostgreSQL: shared_buffers, work_mem and a bloated table that needs VACUUM. See PostgreSQL tuning.

Backups, rsync, tar and find

Large jobs saturate the disk. Run them off-peak, at low priority:

nice -n 19 ionice -c3 /usr/local/bin/backup.sh
rsync --bwlimit=20M …

Prefer incremental backups over full copies every night. See process priority.

Excessive logging

Debug logging, access logs under bot floods, or apps that log every request in detail. Check which files are growing:

sudo find /var/log -type f -mmin -5 -size +10M -exec ls -lh {} +

Reduce log levels, filter noise, and make sure logrotate works. See logrotate.

fsync-heavy workloads

Databases with sync_binlog=1 and innodb_flush_log_at_trx_commit=1 (safe defaults), and apps that commit thousands of tiny transactions, are slow on storage with high write latency. Batch writes in the application. Relax durability settings only if you understand the data-loss trade-off.

File cache churn

Reading huge files once (backups, log analysis) can evict useful cache. Tools like nocache (nocache tar …) avoid polluting the page cache.

Failing disk (dedicated servers)

A dying drive retries reads, and latency explodes. Check dmesg for I/O errors and SMART data. See dmesg and smartctl.

Noisy neighbours (VPS)

On shared storage, another tenant’s heavy I/O can raise your latency. If your own I/O is modest, iotop shows nothing big, and awaits are still high, contact support with iostat -x output.

Tuning options

  • noatime mount option: avoids writing access times on every read. See fstab.
  • The I/O scheduler: virtual disks usually work best with none or mq-deadline. Check cat /sys/block/vda/queue/scheduler.
  • Dirty page settings (vm.dirty_ratio, vm.dirty_background_ratio): lower values flush more evenly and avoid long stalls on big write bursts. See sysctl.
  • Caching layers: Redis or Memcached for hot data, and page caching for websites, which take reads off the disk.
  • Faster storage: SSD or NVMe make a huge difference for random I/O. See NVMe vs SSD.

Quick reference

Task Command
Confirm vmstat 1 (b, wa, si/so)
Which disk iostat -xz 1
Which process sudo iotop -oPa / pidstat -d 1
Stuck processes ps -eo state,pid,cmd | awk '$1 ~ /D/'
Throttle a job ionice -c3 nice -n19 cmd
History sar -d -p

Related: fio disk benchmark and high CPU usage.

  • Linux
  • Performance
  • Storage