When a program crashes with a fatal signal (segmentation fault, abort), Linux can save a core dump: a snapshot of the process’s memory at the moment of the crash. Developers use it to find exactly where and why the program crashed. On servers, core dumps also need managing: they can be large, and contain sensitive data from memory.

Where do core dumps go?

cat /proc/sys/kernel/core_pattern
  • |/usr/lib/systemd/systemd-coredump …: handled by systemd-coredump, and stored in /var/lib/systemd/coredump/ (common on RHEL, Fedora, and Ubuntu with systemd-coredump installed).
  • |/usr/share/apport/apport …: Ubuntu’s apport crash reporter (writes to /var/crash/).
  • core or core.%p: a file in the process’s working directory.

systemd-coredump and coredumpctl

sudo apt install systemd-coredump      # if not installed
coredumpctl list                       # recent crashes
coredumpctl info 12345                 # details for a PID: signal, executable, stack trace
coredumpctl dump 12345 -o core.12345   # extract the core file
coredumpctl debug 12345                # open it in gdb

coredumpctl info alone often shows a short stack trace, which is enough to report a bug.

Limits are set in /etc/systemd/coredump.conf:

[Coredump]
Storage=external
Compress=yes
MaxUse=1G          # total space for dumps
ProcessSizeMax=2G  # skip processes larger than this

ulimit: allow core files

For processes started from a shell, the core file size limit must be non-zero:

ulimit -c                    # 0 = disabled
ulimit -c unlimited          # enable for this shell

For systemd services, set LimitCORE=infinity in the unit (or a drop-in). Default limits for login sessions come from /etc/security/limits.conf.

Analyse a crash with gdb

sudo apt install gdb
gdb /usr/sbin/myapp core.12345
(gdb) bt                 # backtrace: where it crashed
(gdb) bt full            # with local variables
(gdb) info threads
(gdb) thread apply all bt

Readable function names need debug symbols: install the -dbgsym / -dbg / -debuginfo packages for the program and its libraries, or use debuginfod (Ubuntu and Fedora can fetch symbols automatically).

For language runtimes, use their own tools: Java writes hs_err_pid*.log files and heap dumps, Python shows tracebacks (use faulthandler for segfaults), and PHP-FPM logs “child exited on signal 11” (a segfault, often an extension bug).

Production servers: limit or disable

Core dumps can:

  • fill the disk (a crash loop of a big process can write gigabytes quickly);
  • contain secrets (passwords, keys, user data in memory).

Reasonable settings for servers:

# disable for setuid programs (default on most systems)
sysctl fs.suid_dumpable           # 0 is safest

# keep systemd-coredump, but cap its storage (coredump.conf: MaxUse=)

To disable entirely:

# /etc/systemd/coredump.conf
[Coredump]
Storage=none
ProcessSizeMax=0
# /etc/security/limits.conf
* hard core 0

See sysctl tuning.

Clean up

sudo du -sh /var/lib/systemd/coredump /var/crash 2>/dev/null
sudo rm /var/crash/*.crash               # apport reports you no longer need
sudo journalctl --vacuum-time=7d         # coredump metadata lives in the journal

Also look for stray core or core.* files in application directories. See disk full troubleshooting.

Troubleshooting crashes without dumps

  • journalctl -k | grep -i segfault: the kernel logs segfaults, with the program and library.
  • dmesg | tail: OOM kills look like crashes, but aren’t. See Linux OOM killer.
  • Service restart loops: systemctl status svc, journalctl -u svc.

Quick reference

Task Command
Where dumps go cat /proc/sys/kernel/core_pattern
List crashes coredumpctl list
Crash details coredumpctl info <pid>
Debug coredumpctl debug <pid> / gdb prog core
Enable in shell ulimit -c unlimited
Limit storage MaxUse= in coredump.conf

Related: Linux log files explained and Linux kernel panic.

  • Linux
  • Debugging
  • systemd