Master Linux Disk I/O Monitoring with iostat — A Quick, Practical Guide

Effective disk I/O monitoring is essential for maintaining performant Linux servers, especially for sites and applications that rely on consistent read/write throughput—databases, caching layers, and file-serving workloads. Among the standard tools available to Linux administrators, iostat stands out for its simplicity, low overhead, and detailed per-device and per-CPU statistics. This article provides a practical, technically detailed guide to mastering disk I/O monitoring with iostat: how it works, how to interpret its output, realistic use cases, comparisons with other tools, and guidance for selecting VPS hardware based on observed I/O behavior.

How iostat works: fundamentals and metrics

iostat is part of the sysstat package and reads kernel-provided statistics exposed via /proc and /sys to report CPU and block device I/O activity. It summarizes cumulative counters since boot and can display deltas over sampling intervals. Understanding what each metric represents is critical to accurate diagnosis.

Key device fields and what they mean

• Device: block device name (e.g., sda, nvme0n1).
• tps: transactions per second — the number of I/O operations issued to the device per second. Note: for modern NVMe SSDs, a single I/O can represent multiple logical blocks; treat tps as an operation count, not bytes.
• kB_read/s, kB_wrtn/s: throughput in kilobytes per second for read and write operations.
• kB_read, kB_wrtn: cumulative read/write kilobytes since boot (shown with no interval).
• await: average time (ms) for I/O requests issued to the device to be served (includes queuing + service time). High await suggests slow I/O or long queues.
• svctm: average service time (ms) for I/O requests (does not include queueing). Note: on modern kernels and complex stacking (e.g., device-mapper, mdraid), svctm may be unreliable; use await + queue stats for better insight.
• %util: percentage of CPU time during which the device was active servicing requests. Near 100% indicates the device is saturated (throughput or IOPS limited).

Interpreting these numbers together is essential. For example, high %util with high await indicates device saturation; low %util with high await might indicate contention upstream (e.g., virtualization layer, host bus bottleneck) or long individual operations (e.g., large sequential reads).

Practical usage: commands, sampling, and examples

Install sysstat if not present: sudo apt-get install sysstat or yum install sysstat. Basic invocation:

• iostat -x 1 5 — extended device stats sampled every 1 second, 5 times.
• iostat -x -k 5 — kilobytes per second, extended stats, default continuous reporting.
• iostat -dx 2 10 — device-level extended stats, every 2s for 10 samples (useful for capturing spikes).

Sample output snippet (truncated):

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 12.00 5.00 20.00 512.00 1024.00 96.00 0.05 3.10 1.76 4.50

Interpretation:

• tps ~ 25 ops/s (r/s + w/s).
• Throughput ~1.5 MB/s (rkB/s + wkB/s = 1536 kB/s).
• avgqu-sz is average queue length; values >1 suggest a queue forming. If avgqu-sz grows and %util approaches 100, backlog is building.
• await 3.1 ms — acceptable for HDDs; for SSD/NVMe you’d expect <1 ms for low-latency devices under light load.

Diagnosing common conditions

• High %util, high await: device saturated. Options: increase IOPS/throughput capacity (better disk, RAID configuration), offload reads to cache, or spread load across devices.
• Low %util, high await: not device-limited; look at host CPU, virtualization layer or file system locks. Check iostat -c for CPU steal time (%steal).
• High read/write throughput but low tps: indicates large I/O sizes (large sequential ops). Check avgreq-sz (avgrq-sz) to confirm.
• High rrqm/wrqm: many merged requests — the kernel is combining adjacent I/Os; typical for sequential patterns.

Advanced interpretation: kernel internals and virtualization

iostat draws data from kernel block statistics. In virtualized environments (KVM, OpenVZ, Xen), the reported numbers can be influenced by hypervisor scheduling. Pay attention to these factors:

• %steal (reported by iostat when using CPU stats) — CPU cycles taken by the hypervisor. High %steal can indirectly cause higher I/O latency because processes wait for CPU to issue I/O syscalls.
• Disk virtualization layers (virtio, scsi, NVMe passthrough) add complexity: svctm may not reflect true device service time due to request requeuing across layers.
• For NVMe, iostat may report the namespace device (nvme0n1); use iostat -x -p ALL to include partitions and individual namespaces where supported.

Combine iostat with other kernel-provided tools for deeper insight:

• vmstat — shows I/O wait, runnable processes, and swapping behavior.
• iostat -x plus sar -d for historical analysis when sysstat logging is enabled.
• iotop — shows per-process I/O bandwidth in real time (requires kernel accounting support).
• blktrace and btt — for detailed block I/O tracing and latency breakdowns (advanced troubleshooting).

When to use iostat vs other monitoring tools

iostat is ideal for quick, low-overhead snapshots and for scripting periodic checks. It excels when you need device-level metrics without installing heavy agents. However, other tools complement or replace it depending on needs:

• iotop — best for identifying which processes are causing I/O; iostat cannot attribute I/O to PIDs.
• atop — provides long-term process- and resource-level accounting, including disk, network, and per-process metrics.
• collectd/Prometheus node_exporter — for continuous, aggregated monitoring with retention, alerting, and dashboards.
• blktrace — when you need sub-millisecond tracing of block-level events for in-depth performance analysis.

Application scenarios and recommended approaches

Below are typical scenarios with actionable suggestions informed by iostat findings.

Database servers (MySQL, PostgreSQL)

• Look for sustained %util near 100% and rising await — indicates need for faster storage or more IOPS. Consider moving data onto NVMe or provisioning more IOPS if using cloud block storage.
• High write rates with high await recommend tuning fsync/commit settings cautiously, adding battery-backed write cache, or using replicas for read scaling.
• Use iostat sampling during heavy query loads to measure real-world requirements and size storage accordingly.

Web servers and file storage

• Static file serving produces high read throughput but low tps; large avgrq-sz indicates sequential reads — good for throughput-optimized disks.
• For many small file operations (high tps, low kB/s), prioritize IOPS over raw throughput — choose SSD/NVMe or VPS plans that include high IOPS guarantees.

Mixed workloads

• When both random small I/O and large sequential transfers occur, monitor per-device queue lengths (avgqu-sz) and consider separating workloads onto different disks or volumes.

Choosing VPS hardware based on iostat insights

When selecting a VPS plan or upgrading, map observed iostat metrics to hardware requirements:

• If you see high sustained throughput (MB/s) with moderate tps: prioritize high sequential bandwidth — choose NVMe with good throughput guarantees and a strong host network for remote storage access.
• If you see high tps with low throughput (many small I/Os): prioritize IOPS and low latency — NVMe with high IOPS per instance or dedicated SSD-backed volumes is preferable.
• If %util is high and CPU steal (%steal) is elevated, look for VPS providers with dedicated CPU or less noisy neighbors — CPU contention at the host can affect I/O performance.
• For redundancy and availability, evaluate RAID options, snapshots, and backup performance — ensure storage snapshots do not spike I/O at critical times.
• Request or test realistic workloads (benchmarks like fio) on candidate VPSs to validate provider claims for IOPS and latency. Synthetic tests combined with iostat during test runs give measured expectations for production.

Quick tuning checklist

• Enable write caching carefully and only with reliable power protection or provider guarantees.
• Tune filesystem mount options (e.g., noatime, barrier settings) based on application durability requirements.
• For databases, adjust checkpointing and journaling parameters to smooth I/O bursts.
• Consider using tmpfs for ephemeral, high-I/O temporary data to offload disk.
• Monitor regularly: set up historical collection with sar or Prometheus to detect regressions early.

Summary

iostat is a lightweight, powerful first-line tool for diagnosing disk I/O performance on Linux. By understanding its fields—tps, throughput, await, svctm, and %util—and combining iostat with complementary tools like iotop, vmstat, and blktrace, you can quickly identify whether problems stem from device saturation, host contention, or application behavior. Use realistic sampling intervals to capture bursts, and translate observed metrics into concrete infrastructure decisions: higher IOPS, NVMe storage, dedicated CPU, or workload separation.

If you’re planning capacity changes or evaluating VPS options based on I/O demands, run representative tests during evaluation and match the provider’s disk characteristics to your observed needs. For example, consider provider plans that offer NVMe-backed instances with consistent IOPS and low latency. You can learn more about suitable options and check availability at VPS.DO — for readers in the U.S., the USA VPS offerings provide a range of SSD and NVMe-backed configurations worth testing with your typical workloads.