top, vmstat and iostat show what’s happening now. But slowdowns are often reported afterwards: “the site was slow last night around 3am”. sar (system activity reporter, part of sysstat) records system statistics every few minutes and lets you look back days later. It’s lightweight and works without any monitoring server.

Install and enable collection

sudo apt install sysstat        # Debian/Ubuntu
sudo dnf install sysstat        # RHEL family

On Debian/Ubuntu, enable data collection:

sudo sed -i 's/^ENABLED="false"/ENABLED="true"/' /etc/default/sysstat
sudo systemctl enable --now sysstat

On RHEL-family systems, enabling the service (sysstat plus sysstat-collect.timer) is enough.

By default, a sample is taken every 10 minutes and kept for a week to a month, depending on the distribution’s HISTORY setting in /etc/sysstat/sysstat (or /etc/sysconfig/sysstat). Data files:

  • Debian/Ubuntu: /var/log/sysstat/saDD
  • RHEL family: /var/log/sa/saDD

DD is the day of the month.

For finer resolution, change the collection interval with a timer override:

sudo systemctl edit sysstat-collect.timer
# [Timer]
# OnCalendar=
# OnCalendar=*:00/2

That gives a sample every 2 minutes.

Read today’s data

sar                 # CPU (default)
sar -r              # memory
sar -S              # swap
sar -W              # swapping activity (pages in/out per second)
sar -q              # run queue and load average
sar -d -p           # disks, with readable device names
sar -n DEV          # network traffic per interface
sar -n EDEV         # network errors
sar -A              # everything

Look at another day or time window

sar -f /var/log/sysstat/sa14                    # the 14th
sar -q -f /var/log/sysstat/sa14 -s 02:00:00 -e 04:00:00   # 2am–4am on the 14th
sar -1                                          # yesterday (newer sysstat)

What to look for

CPU (sar):

02:50:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
03:00:01 AM     all     12.31      0.00      3.10     41.52      0.20     42.87

High %iowait at 3am points to disk activity. Backups or a cron job are the usual suspects. %steal shows hypervisor contention on a VPS.

Load (sar -q): runq-sz (runnable processes) and ldavg-1/5/15. blocked shows processes waiting on I/O. See load average.

Memory (sar -r): %memused includes cache, so look at kbavail (available memory) and %commit. Combine it with sar -W: non-zero pswpin/s and pswpout/s means swapping.

Disk (sar -d -p): await (latency in ms) and %util per device, the same meaning as in iostat.

Network (sar -n DEV): rxkB/s and txkB/s per interface. Sudden spikes can mean a traffic surge, a large backup upload or an attack.

A real investigation

“Website slow every night around 03:00”:

sar -q -s 02:30:00 -e 03:30:00          # load jumps at 03:00?
sar -s 02:30:00 -e 03:30:00             # iowait jumps?
sar -d -p -s 02:30:00 -e 03:30:00       # which disk, what latency?
grep -rE '^[^#].* (3|03) ' /etc/crontab /etc/cron.d/ ; sudo crontab -l    # what runs at 3am?
systemctl list-timers                    # systemd timers too

Typical result: a backup or updatedb job saturates the disk. Fix it by moving the job, throttling it with ionice -c3 nice -n19, or making the backup incremental. See crontab.

Export for graphs

sadf converts sar data to CSV, JSON or SVG:

sadf -d /var/log/sysstat/sa14 -- -r > mem-14.csv        # CSV for a spreadsheet
sadf -g /var/log/sysstat/sa14 -- -q > load-14.svg       # SVG graph

Other sysstat tools

Tool Shows
iostat Disk I/O, live
mpstat -P ALL 1 Per-CPU-core usage
pidstat 1 Per-process CPU (-r memory, -d disk)
sar History of everything

sar vs a monitoring stack

sar is perfect for a single server, or as a fallback that keeps working when the monitoring server is down. For dashboards, alerts and many servers, use Netdata or Prometheus and Grafana. Many admins run both.

Quick reference

Task Command
Enable ENABLED="true" + systemctl enable --now sysstat
CPU today sar
Load / run queue sar -q
Memory / swapping sar -r / sar -W
Disks sar -d -p
Network sar -n DEV
Other day sar -f /var/log/sysstat/saDD
Time window -s 02:00:00 -e 04:00:00

Related: vmstat and top.

  • Linux
  • Performance
  • Monitoring