top tells you which process is using the CPU. perf tells you which functions inside it are using it. It’s the Linux kernel’s built-in profiler: it samples what the CPU is executing many times per second, and shows you the hotspots, for the kernel and user programs alike. Combined with flame graphs, it turns “the server is slow” into “this function in this library is the problem”.

Install

sudo apt install linux-tools-common linux-tools-$(uname -r)     # Ubuntu
sudo apt install linux-perf                                    # Debian
sudo dnf install perf                                          # RHEL/AlmaLinux
perf --version

Live view: perf top

sudo perf top                 # system-wide hotspots, like top for functions
sudo perf top -p $(pgrep -o mysqld)

You’ll see functions sorted by the share of CPU samples, with the library or binary they belong to ([kernel], libc.so, mysqld). Hex addresses instead of names mean symbols are missing (see below).

Record and analyse

sudo perf record -F 99 -g -a -- sleep 30          # whole system, 30 s, with call graphs
sudo perf record -F 99 -g -p 1234 -- sleep 30     # one process
sudo perf report                                   # interactive report
sudo perf report --stdio | head -50
  • -F 99: sample 99 times per second (low overhead, and avoids lock-step with timers).
  • -g: record call stacks, so you see why a function is called.
  • The data is written to perf.data in the current directory.

Flame graphs

Flame graphs visualise stack samples: width = time spent, and the stack grows upward. They make it easy to spot the widest (most expensive) code paths.

git clone https://github.com/brendangregg/FlameGraph
sudo perf record -F 99 -g -a -- sleep 30
sudo perf script | ./FlameGraph/stackcollapse-perf.pl | ./FlameGraph/flamegraph.pl > cpu.svg

Open cpu.svg in a browser (copy it to your computer with scp). You can click to zoom into a frame. See scp command.

Symbols and language runtimes

  • Compiled programs: install debug symbol packages (-dbg / -dbgsym / debuginfo) for readable function names.
  • Missing frames: programs compiled without frame pointers produce broken stacks. Try perf record --call-graph dwarf (larger data), or use builds with frame pointers (enabled by default in recent Ubuntu and Fedora releases).
  • Java: use perf-map-agent or async-profiler to map JIT-compiled methods.
  • Node.js: run with --perf-basic-prof to emit a perf map.
  • Python 3.12+: python -X perf enables perf support. For older versions, use py-spy (py-spy top --pid N, py-spy record -o out.svg --pid N).
  • PHP: use php-spx, Xdebug’s profiler (on staging), or Blackfire/Tideways, which understand PHP functions.

Other useful perf commands

sudo perf stat -a -- sleep 10           # counters: context switches, CPU migrations, page faults
sudo perf stat -e cache-misses,instructions,cycles -p 1234 -- sleep 10
sudo perf trace -p 1234                 # syscall trace (like strace, lower overhead)

On a VPS: caveats

  • Hardware performance counters (cycles, cache misses) are often not available inside virtual machines. perf stat then shows “not supported”. Software events and timer-based sampling (perf record -F 99, perf top) still work.
  • Some kernels restrict perf for non-root users via kernel.perf_event_paranoid. Run as root, or lower it temporarily (sudo sysctl kernel.perf_event_paranoid=1).
  • High steal time means the hypervisor gives your VM less CPU than it wants, and no profiling inside the VM fixes that. See VPS performance troubleshooting.

A practical workflow

  1. top/htop: which process is hot? See htop.
  2. perf top -p PID: which functions?
  3. perf record -g + flame graph: which code paths lead there?
  4. Fix (query, cache, configuration, code), then measure again.

Quick reference

Task Command
Live hotspots sudo perf top [-p PID]
Record 30 s with stacks sudo perf record -F 99 -g -a -- sleep 30
Report sudo perf report
Flame graph perf script | stackcollapse-perf.pl | flamegraph.pl > out.svg
Counters sudo perf stat -a -- sleep 10
Python py-spy top --pid N

Related: eBPF basics and Linux CPU usage high.

  • Linux
  • Performance
  • Profiling