Under load, a busy service starts failing with:

socket() failed (24: Too many open files)
accept4() failed (24: Too many open files)
EMFILE: too many open files

Every network connection, file and pipe a process uses takes a file descriptor. Linux caps how many each process may open, often at 1024 by default. That’s far too low for a busy web server, database or Node.js app. Here’s how to see the limits and raise them properly.

Check the current limits

For your shell:

ulimit -n         # soft limit on open files
ulimit -Hn        # hard limit
ulimit -a         # all limits
  • The soft limit is what’s enforced. A process can raise it up to the hard limit.
  • The hard limit is the ceiling. Only root can raise it.

For a running service, which is what matters:

cat /proc/$(pgrep -o nginx)/limits | grep 'open files'
cat /proc/$(pgrep -o mysqld)/limits | grep 'open files'

How many it’s using right now:

sudo ls /proc/$(pgrep -o nginx)/fd | wc -l
sudo lsof -p $(pgrep -o node) | wc -l

System-wide totals:

cat /proc/sys/fs/file-nr         # allocated, unused, maximum
cat /proc/sys/fs/file-max

The system-wide maximum (fs.file-max) is usually huge on modern kernels. The per-process limit is the usual bottleneck.

Raise it for systemd services (the usual fix)

Services started by systemd ignore /etc/security/limits.conf. Set the limit in the unit with a drop-in:

sudo systemctl edit nginx
[Service]
LimitNOFILE=65535
sudo systemctl daemon-reload
sudo systemctl restart nginx
cat /proc/$(pgrep -o nginx)/limits | grep 'open files'

Do the same for mysql/mariadb, php8.3-fpm, redis-server, your Node or Python app units, and docker or containerd.

To change the default for all services, set DefaultLimitNOFILE=65535 in /etc/systemd/system.conf, then run sudo systemctl daemon-reexec. Recent systemd versions already default to a soft limit of 1024 and a hard limit of 524288, which lets well-behaved programs raise their own soft limit.

Raise it for users (interactive and SSH sessions)

For processes started from a login shell, such as scripts or apps run in tmux:

# /etc/security/limits.d/90-nofile.conf
*       soft    nofile  65535
*       hard    nofile  65535
deploy  soft    nofile  100000
deploy  hard    nofile  100000

This is applied by PAM at login. Log out and back in, then check ulimit -n. It does not affect systemd services.

Temporarily, for the current shell (up to the hard limit):

ulimit -n 65535

Application-specific settings

Raising the OS limit isn’t always enough. Some programs have their own setting.

Nginx:

worker_rlimit_nofile 65535;
events {
    worker_connections 16384;
}

Each proxied connection uses two descriptors (client plus upstream), so keep worker_rlimit_nofile at about 2× worker_connections or more.

MySQL/MariaDB:

[mysqld]
open_files_limit = 65535
table_open_cache = 4000

MySQL can’t exceed the systemd LimitNOFILE. Check with SHOW VARIABLES LIKE 'open_files_limit';.

Node.js uses the process limit directly. Raise LimitNOFILE in its unit or PM2’s systemd unit. See systemd service for Node.

Elasticsearch/OpenSearch require at least 65535 and refuse to start otherwise.

Find a descriptor leak

If usage climbs steadily until it hits the limit no matter how high you set it, the app is leaking descriptors: connections or files opened and never closed.

watch -n 5 'sudo ls /proc/$(pgrep -o node)/fd | wc -l'
sudo lsof -p PID | awk '{print $5}' | sort | uniq -c | sort -rn     # by type (IPv4, REG, FIFO…)
sudo lsof -p PID | grep -c ESTABLISHED

Thousands of sockets in CLOSE_WAIT (ss -tanp | grep CLOSE_WAIT) mean the app isn’t closing connections. That’s a code bug, typically an HTTP client or database connection that’s never released. See lsof and ss.

How high is sensible?

65535 is a common, safe value for web servers, proxies and databases. Very high-connection systems (100k+ concurrent sockets) go higher and also tune kernel network settings. See sysctl tuning. Each descriptor uses a little kernel memory, so don’t set millions without a reason.

Quick reference

Task Command / setting
Shell limit ulimit -n / ulimit -Hn
Service limit cat /proc/PID/limits
In use ls /proc/PID/fd | wc -l
systemd service LimitNOFILE=65535 (systemctl edit)
Users /etc/security/limits.d/*.conf
Nginx worker_rlimit_nofile
MySQL open_files_limit

Related: Nginx 502 errors and MySQL too many connections.

  • Linux
  • Performance
  • Troubleshooting