Ubuntu Server Disk Space Full: Causes and Solutions

Running out of disk space on Ubuntu Server is a frequent issue that can halt services, prevent logins, block updates, or cause boot failures. The error often appears as “No space left on device,” even when df -h shows some free space (due to reserved blocks or inode exhaustion). Below are the most common causes on modern Ubuntu Server (24.04 LTS and later) and proven remediation steps.

Common Causes

1. Large or Uncontrolled Log Files (/var/log) Services like nginx/Apache, PostgreSQL, systemd-journald, kernel, or failing hardware (e.g., overheating GPU) generate massive logs. A single syslog or journal file can grow to tens or hundreds of GB if rotation fails or an error loop occurs.
2. APT Package Cache (/var/cache/apt/archives) Downloaded .deb packages accumulate after upgrades or installs. Automatic security updates exacerbate this without cleanup.
3. Old Snap Revisions (/var/lib/snapd/snaps) Snap packages keep multiple revisions for rollback. On servers with many snaps (e.g., Nextcloud, Prometheus), old versions consume significant space.
4. Old Kernel Images (/boot or /) Each kernel upgrade adds ~200–500 MB. Ubuntu retains several versions for safety, but they pile up over time.
5. Deleted Files Still Held Open by Processes A process (e.g., a daemon) keeps a file descriptor open after deletion → space isn’t freed until the process restarts or is killed.
6. Reserved Blocks on ext4 Filesystems By default, ext4 reserves 5% for root only. On small partitions, this creates the illusion of “full” when df shows 0% free for non-root users.
7. Inode Exhaustion Millions of small files (e.g., mail queue spam, temp files, Docker overlay2 layers) deplete inodes even with free space.
8. Other Culprits
   • Docker/Podman images, containers, volumes (/var/lib/docker).
   • Large databases or application data in /var/lib.
   • Journald persistent logs (/var/log/journal).
   • User mailboxes or error logs for root.

Step-by-Step Diagnosis

Start here when space is critically low (system may become read-only).

1. Check Overall Usage
   Bash

   df -h
   df -i   # Check inode usage

2. Find Largest Directories (Quick Scan)
   Bash

   sudo du -h --max-depth=1 / | sort -hr | head -n 15

   Focus on /var, /usr, /home, /boot. Drill deeper:

   Bash

   sudo du -h --max-depth=1 /var | sort -hr

3. Interactive Explorer (Highly Recommended)
   Bash

   sudo apt install ncdu
   sudo ncdu /

   Navigate with arrows, delete with ‘d’ — perfect for servers.

4. Check Specific Hotspots
   • Logs: sudo du -sh /var/log/* | sort -hr
   • APT cache: sudo du -sh /var/cache/apt
   • Snaps: snap list –all (look for disabled revisions)
   • Journal: sudo journalctl –disk-usage
   • Deleted-but-open files: sudo lsof +L1 or sudo ls -l /proc/*/fd/* | grep deleted

Immediate Relief Actions (Safe First)

• Clean APT cache:
  Bash

  sudo apt clean
  sudo apt autoclean

• Remove old kernels (safe on stable systems):
  Bash

  sudo apt autoremove --purge

• Clear snap old revisions:
  Bash

  sudo snap list --all | awk '/disabled/{print $1, $3}' | while read snapname revision; do sudo snap remove "$snapname" --revision="$revision"; done

• Truncate huge logs (if identified):
  Bash

  sudo truncate -s 0 /var/log/syslog
  sudo truncate -s 0 /var/log/kern.log

• Vacuum journald:
  Bash

  sudo journalctl --vacuum-time=2weeks
  sudo journalctl --vacuum-size=500M

• Restart processes holding deleted files (after identifying via lsof).

Prevention & Best Practices

• Schedule cleanups via cron or systemd timers.
• Configure logrotate aggressively for high-log services.
• Set journald limits in /etc/systemd/journald.conf (e.g., SystemMaxUse=500M).
• Use tmpfs for /tmp if RAM allows.
• Monitor proactively: Netdata, Prometheus Node Exporter, or simple cron job emailing df -h.
• On small root partitions: Reduce ext4 reserved blocks (sudo tune2fs -m 1 /dev/sdX).
• For production: Separate /var, /home, or data volumes.

Quick Summary Table

If the system is completely unusable (no login possible), boot from live USB, mount the root filesystem, and clean from there.

Share output from df -h, du -h –max-depth=1 / | sort -hr, or your workload type (web server, database, containers, etc.), and more tailored fixes can be provided.