Your application fails with No space left on device, but df -h shows gigabytes free. The usual culprit is inodes. The filesystem has run out of entries for new files, even though there’s room for data.

What an inode is

On ext4, every file and directory needs one inode, a small record holding its metadata (owner, permissions, timestamps, where the data blocks are). The number of inodes is fixed when the filesystem is created, typically one per 16 KiB of disk. Millions of tiny files can use every inode while most of the space stays empty.

XFS and Btrfs allocate inodes dynamically, so this problem is almost always ext4.

Confirm it

df -i
Filesystem      Inodes   IUsed  IFree IUse% Mounted on
/dev/vda1      5242880 5242880      0  100% /

IUse% 100% means out of inodes. Compare with df -h, which shows plenty of free space.

Find where the files are

Count inodes per directory, starting at the top (GNU du 8.22+):

sudo du --inodes -x -d 1 / 2>/dev/null | sort -n | tail -15

Then drill into the biggest one:

sudo du --inodes -x -d 1 /var 2>/dev/null | sort -n | tail -15
sudo du --inodes -x -d 1 /var/lib/php 2>/dev/null | sort -n | tail

Alternative with find, counting files per directory:

sudo find / -xdev -type f -printf '%h\n' 2>/dev/null | sort | uniq -c | sort -rn | head -20

The usual suspects

Location Cause
/var/lib/php/sessions PHP session files never cleaned up (the session GC is disabled on Debian/Ubuntu and relies on a cron job)
/var/spool/postfix/maildrop, /var/spool/exim4/input Stuck mail queue, often from a script sending mail that can’t be delivered
/var/spool/clientmqueue Cron output sent through sendmail with no working MTA
App cache dirs (storage/framework/cache, wp-content/cache) File-based caches with no expiry
/tmp Temporary files from scripts or uploads
node_modules, vendor in many releases Old deployment releases never cleaned up
Docker overlay2 Old images and containers. See docker prune
Backup tools Snapshot-style backups with millions of hard links

Clean up

Deleting millions of files with rm * fails with “Argument list too long”. Use find:

# PHP sessions older than a day
sudo find /var/lib/php/sessions -type f -mmin +1440 -delete

# app cache
sudo find /var/www/app/storage/framework/cache -type f -delete

# old temp files
sudo find /tmp -type f -mtime +7 -delete

Stuck mail queue (Postfix): inspect it first with mailq | tail, fix the sender, then:

sudo postsuper -d ALL

clientmqueue: delete the files, then fix cron jobs so they don’t produce mail. Redirect their output with >/dev/null 2>&1, or set MAILTO="".

Huge directories delete faster with rsync:

mkdir /tmp/empty && sudo rsync -a --delete /tmp/empty/ /path/to/huge-dir/

Check df -i again afterwards.

Prevent it

  • Fix the source. Configure session cleanup, expire caches, repair the mail setup, prune old releases.
  • Monitor inodes, not just space. Most monitoring tools (Netdata, node_exporter) track inode usage. Alert at 80%.
  • Use Redis for sessions and cache instead of files. See WordPress Redis object cache.
  • Choose the filesystem for the workload. For a disk that will hold millions of small files, create ext4 with more inodes (mkfs.ext4 -i 4096 or -T small), or use XFS. The inode count of an existing ext4 filesystem can’t be increased without reformatting. See mkfs.
  • Schedule cleanups with systemd-tmpfiles or cron:
# /etc/cron.daily/cleanup-cache
find /var/www/app/storage/framework/cache -type f -mtime +2 -delete

Other causes of “No space left on device”

If neither space nor inodes are exhausted:

  • Deleted files still held open by a process. The space isn’t freed until the process closes them. Find them with sudo lsof +L1 and restart that service. See lsof.
  • Reserved blocks: ext4 keeps 5% for root, so normal users see “full” earlier.
  • Quotas for the user or project.
  • A different filesystem is full: /tmp, /boot or a Docker volume. Check df -h and df -i for every mount.

See disk full troubleshooting.

Quick reference

Task Command
Inode usage df -i
Inodes per directory du --inodes -x -d 1 /path | sort -n
Files per directory find / -xdev -type f -printf '%h\n' | sort | uniq -c | sort -rn | head
Bulk delete find dir -type f -mtime +N -delete
Deleted but open lsof +L1

Related: du and find large files.

  • Linux
  • Storage
  • Troubleshooting