Every open file, socket and pipe a process uses is represented by a file descriptor (FD), a small integer. Three are open by default in every process. Once you understand them, shell redirection (>, 2>&1, |) makes sense, and so do errors like “Too many open files”.

The three standard descriptors

FD Name Default
0 stdin (standard input) Keyboard / terminal
1 stdout (standard output) Terminal
2 stderr (standard error) Terminal

Programs write normal output to 1 and error messages to 2. Both show up on your terminal, but they’re separate streams, and you can redirect them independently.

Redirecting output

command > out.txt          # stdout to file (overwrite)
command >> out.txt         # stdout to file (append)
command 2> err.txt         # stderr to file
command > out.txt 2> err.txt
command > all.txt 2>&1     # both to the same file
command &> all.txt         # same, Bash shorthand
command &>> all.txt        # append both (Bash 4+)

Order matters with 2>&1

2>&1 means “make FD 2 point wherever FD 1 points right now”:

command > file 2>&1        # correct: stdout→file, then stderr→(same as stdout)=file
command 2>&1 > file        # wrong: stderr→terminal (where stdout was), then stdout→file

This is one of the most common shell mistakes. It’s why cron jobs “lose” error messages.

Discarding output: /dev/null

command > /dev/null             # hide normal output
command 2> /dev/null            # hide errors (e.g. "Permission denied" from find)
command > /dev/null 2>&1        # silence everything

Typical uses: find / -name x 2>/dev/null, and cron jobs where you only care about the exit code. Don’t hide errors from important cron jobs. Log them instead:

0 3 * * * /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1

Pipes connect stdout to stdin

grep error app.log | sort | uniq -c

A pipe only carries stdout. To pipe errors as well:

command 2>&1 | grep -i error
command |& grep -i error         # Bash shorthand

Input redirection

mysql shop < dump.sql            # file as stdin
wc -l < access.log
while read -r line; do …; done < list.txt
cat <<'EOF' > file.txt           # here-document
text
EOF
grep x <<< "$variable"           # here-string

See cat and heredocs and tee.

Errors to stderr in your scripts

echo "ERROR: config missing" >&2

>&2 sends your message to stderr, so it isn’t mixed into data that’s piped or captured with $(…).

Opening your own descriptors

Bash can open extra FDs (3–9 are free to use):

exec 3> /var/log/myscript.log     # open FD 3 for writing
echo "starting" >&3
exec 3>&-                         # close it

exec 9> /run/lock/myjob.lock      # common pattern for flock
flock -n 9 || exit 1

Inspecting open descriptors

Every process’s descriptors are visible in /proc:

ls -l /proc/$$/fd                         # your shell
sudo ls -l /proc/$(pgrep -o nginx)/fd | head
sudo ls /proc/PID/fd | wc -l              # how many are open
lr-x------ 1 root root 64 … 0 -> /dev/null
l-wx------ 1 root root 64 … 1 -> /var/log/nginx/access.log
lrwx------ 1 root root 64 … 6 -> socket:[123456]

lsof -p PID shows the same with more detail: files, sockets with addresses, pipes. See lsof.

This is how you find:

  • which log file a process is really writing to (useful after logrotate problems);
  • deleted files still held open. They show (deleted) and still use disk space. Restart the process to free it;
  • descriptor leaks, where the count keeps rising until “Too many open files”. See ulimit and open files.

Recover a deleted file that’s still open

If a running process still has a deleted file open, its contents are still readable:

sudo ls -l /proc/PID/fd | grep deleted
sudo cp /proc/PID/fd/4 /tmp/recovered.log

Everything is a file (descriptor)

Sockets, pipes, terminals, devices, inotify watches and timers all use FDs. That’s why a busy web server needs thousands of them, and why the per-process limit (often 1024 by default) matters so much.

Quick reference

Syntax Meaning
> f / >> f stdout to file (overwrite / append)
2> f stderr to file
> f 2>&1 / &> f both to file
2>/dev/null discard errors
| / |& pipe stdout / stdout+stderr
< f file as stdin
>&2 write to stderr
ls -l /proc/PID/fd a process’s open FDs

Related: Bash scripting and crontab.

  • Linux
  • Bash
  • Basics