MySQL stopped in the middle of the night. Your Node app vanished with no error in its own log. Docker containers restart with exit code 137. In all of these cases, the likely culprit is the kernel’s OOM (Out Of Memory) killer. When RAM and swap are exhausted, the kernel picks a process and kills it to keep the system alive.

Confirm it was the OOM killer

sudo dmesg -T | grep -iE 'out of memory|oom-kill|killed process'
sudo journalctl -k --since "24 hours ago" | grep -i 'killed process'
Out of memory: Killed process 2311 (mysqld) total-vm:2516432kB, anon-rss:1621340kB, file-rss:0kB, shmem-rss:0kB, UID:110 pgtables:3540kB oom_score_adj:0
  • Killed process — the victim and its PID.
  • anon-rss — how much memory it actually used.

Above that line, the kernel prints a table of every process and its memory use at the moment of the kill. That’s very useful to see what else was eating RAM. Check the previous boot too (journalctl -k -b -1) if the server rebooted.

For containers: docker inspect --format '{{.State.OOMKilled}}' name shows true, and the exit code is 137. Systemd services show oom-kill in systemctl status.

How the victim is chosen

Each process has an oom_score (0–1000), based mostly on its memory use, adjusted by oom_score_adj (−1000 to +1000). The highest score is killed. That’s why the largest process, usually the database, is the typical victim, even if a runaway PHP worker caused the shortage.

for p in $(pgrep -d' ' 'mysqld|node|php-fpm|java'); do
  printf '%-8s %-20s %s\n' "$p" "$(cat /proc/$p/comm)" "$(cat /proc/$p/oom_score)"
done

Find out what’s using memory

free -h
ps -eo pid,user,rss,cmd --sort=-rss | head -15       # RSS in KiB

See free and ps. Common causes on a VPS:

Cause Fix
PHP-FPM pm.max_children too high Calculate it from RAM. See PHP-FPM tuning
MySQL buffer pool too large for the RAM Size innodb_buffer_pool_size to leave room. See buffer pool
Memory leak in an app (Node, Python, Java) Fix the leak. Meanwhile, restart on high memory (PM2 max_memory_restart, systemd limits)
Java heap not capped Set -Xmx
Too many services for the plan size Move some elsewhere, or upgrade RAM
Traffic spikes or bots multiplying workers Caching, rate limiting

Prevention

1. Add swap as a buffer

Even a small swap file (1–2 GB) gives the kernel room to move idle memory out, instead of killing something immediately during a spike:

sudo fallocate -l 2G /swapfile && sudo chmod 600 /swapfile
sudo mkswap /swapfile && sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

Swap is a safety net, not extra RAM. Constant swapping makes everything slow. See swap usage.

2. Protect critical services

Make the database a less likely victim:

sudo systemctl edit mysql
[Service]
OOMScoreAdjust=-600

Don’t use −1000 (never kill) on large services. The kernel then kills everything else, sshd included, and you may lose access.

3. Cap memory for less important services

systemd uses cgroups to contain a service. If it exceeds its limit, only that service is killed or throttled:

[Service]
MemoryHigh=700M      # throttle and reclaim above this
MemoryMax=800M       # hard limit: OOM kill within this service only

Docker:

docker run -m 512m --memory-swap 512m …

or deploy.resources.limits.memory in Compose. See Docker resource limits.

4. Earlier, gentler intervention: systemd-oomd / earlyoom

Ubuntu 22.04+ ships systemd-oomd, which acts on memory pressure before the kernel’s last-resort OOM killer, and kills whole cgroups more predictably. earlyoom is a simple alternative for other distributions.

5. Monitor memory

Alert when available memory stays below ~10–15%, or swap activity is continuous. Tools: Netdata, Prometheus node_exporter, or sar -r history (sar).

Right-size the server

If the OOM killer strikes regularly even after tuning, the workload simply needs more memory. A practical rule: steady-state usage (excluding cache) should leave 20–30% headroom for spikes. VPS.DO plans range from 2 GB to 40 GB RAM, and upgrades are handled through a support ticket. See how much RAM does a VPS need.

Quick reference

Task Command
Find OOM kills dmesg -T | grep -i 'killed process'
Previous boot journalctl -k -b -1 | grep -i oom
Top memory users ps -eo pid,rss,cmd --sort=-rss | head
Process score cat /proc/PID/oom_score
Protect a service OOMScoreAdjust=-600
Cap a service MemoryMax=800M
Container OOM? docker inspect -f '{{.State.OOMKilled}}' name

Related: vmstat and load average.

  • Linux
  • Performance
  • Troubleshooting