When a program fails with a vague error, hangs, or just “doesn’t work”, and its logs say nothing useful, strace shows what it’s actually doing. It prints every system call the process makes: each file it tries to open, each network connection, each read and write, and the result. It’s the tool that answers “which config file is it reading?” and “what is it waiting for?”

Install

sudo apt install strace       # Debian/Ubuntu
sudo dnf install strace       # RHEL family

Trace a command

strace ls /nonexistent
execve("/usr/bin/ls", ["ls", "/nonexistent"], …) = 0
…
statx(AT_FDCWD, "/nonexistent", …) = -1 ENOENT (No such file or directory)
write(2, "ls: cannot access '/nonexistent'…", 61) = 61
exit_group(2) = ?

Each line has the form syscall(arguments) = return value. Errors return -1 with a code like ENOENT (not found), EACCES (permission denied) or ECONNREFUSED.

Output is long, so write it to a file:

strace -o /tmp/trace.txt -f command args

The most useful options

strace -f cmd               # follow child processes and threads (almost always want this)
strace -p PID               # attach to a running process
strace -e trace=file cmd    # only file-related calls (open, stat, access…)
strace -e trace=network cmd # only network calls
strace -e trace=%process cmd   # fork/exec/exit
strace -e openat,connect cmd   # specific calls
strace -tt -T cmd           # timestamps and time spent in each call
strace -c cmd               # summary: count and time per syscall
strace -s 200 cmd           # show longer strings (default 32 chars)
strace -y cmd               # show file paths for descriptors

Practical recipes

Which files does it try to open?

strace -f -e trace=openat,stat,access -o /tmp/t.txt php artisan config:cache
grep -E 'ENOENT|EACCES' /tmp/t.txt

That instantly shows a missing config, a wrong include path or a permission problem, with the exact path the program tried.

Why permission denied?

strace -f -e trace=file nginx -t 2>&1 | grep EACCES

If Unix permissions look right but EACCES appears, think of SELinux or AppArmor, or a missing x bit on a parent directory. See SELinux and AppArmor.

What’s a hung process waiting for?

sudo strace -p 12345
  • Stuck on read(5, … or recvfrom → waiting for data from a socket or pipe. Find FD 5 with ls -l /proc/12345/fd/5.
  • Stuck on connect(…) → a network connection that isn’t answering (a firewall drop, or a dead database host).
  • Stuck on futex(…) → waiting on a lock between threads.
  • Repeating poll/epoll_wait with timeouts → idle event loop. That’s normal for servers.

Where does the time go?

strace -c -f ./slow-script.sh

The summary table shows which syscalls take the most time, for example thousands of stat calls (autoloaders searching paths) or slow fsync.

strace -f -tt -T -e trace=network -p $(pgrep -o php-fpm)

-T shows the duration of each call in <0.523411> brackets. Long connect or recvfrom times point to slow external services.

Trace PHP-FPM or a web request

Attach to all workers of a pool while you reproduce the request:

sudo strace -f -tt -s 200 -o /tmp/fpm.txt $(pgrep php-fpm | sed 's/^/-p /')

Then search for your script name and errors in the output.

Cautions

  • Performance: strace slows the traced process down considerably, sometimes 10× or more. Avoid long traces on busy production processes, and trace briefly and with filters.
  • Sensitive data: traces can contain passwords, tokens and request bodies (-s shows more). Delete trace files afterwards.
  • Permissions: attaching to other users’ processes needs root. Hardened systems may restrict ptrace (kernel.yama.ptrace_scope).
  • Containers: trace from the host using the host PID, or run strace inside a container with the SYS_PTRACE capability.
Tool Use
ltrace Library calls instead of system calls
lsof -p PID Open files and sockets (snapshot)
perf trace Lower-overhead syscall tracing
bpftrace / bcc tools (opensnoop, execsnoop) System-wide tracing with minimal overhead
gdb Full debugger, stack traces

Quick reference

Task Command
Trace command with children strace -f -o out.txt cmd
Attach to PID strace -p PID
Files only -e trace=file
Network only -e trace=network
Timing -tt -T
Summary strace -c cmd
Longer strings -s 200
Find errors grep -E 'ENOENT|EACCES' out.txt

Related: lsof and /proc filesystem.

  • Linux
  • Troubleshooting
  • Debugging