Bash Error Handling: set -euo pipefail, trap and Exit Codes

By default, Bash keeps going after a command fails. A backup script whose mysqldump fails will happily compress an empty file, upload it and delete yesterday’s good backup. Good error handling makes scripts stop on failure, clean up, and tell you what went wrong.

Exit codes

Every command returns an exit status: 0 for success and 1–255 for failure. Check the last one with $?:

grep -q nginx /etc/passwd
echo $?        # 0 if found, 1 if not, 2 on error

Your scripts should exit non-zero on failure, so cron, CI and systemd notice:

exit 1

The strict-mode header

#!/usr/bin/env bash
set -euo pipefail

Option	Effect
`-e` (errexit)	Exit when a command fails
`-u` (nounset)	Treat unset variables as errors
`-o pipefail`	A pipeline fails if any command in it fails, not just the last one

Without pipefail, mysqldump shop | gzip > shop.sql.gz “succeeds” whenever gzip succeeds, even if the dump failed.

Add -x temporarily to print each command before it runs, for debugging: bash -x script.sh.

set -e pitfalls

set -e is helpful, but it has exceptions you need to know about.

Commands in conditions don’t trigger it:

if grep -q x file; then …; fi    # failure is normal here
cmd || true                      # explicitly allow failure
cmd && other                     # failure of cmd doesn't exit

Commands that “fail” normally. grep with no match returns 1, and so does diff with differences, and ((count++)) when count is 0. Under set -e these exit your script. Guard them:

matches=$(grep -c ERROR app.log || true)
(( count++ )) || true      # or: count=$((count + 1))

local var=$(cmd) hides failures, because the status is that of local. Split it: local var; var=$(cmd).

Functions called in a condition lose set -e inside them entirely: if myfunc; then. Handle errors explicitly in such functions.

Because of these rules, treat set -e as a safety net, not a replacement for checking important commands.

Check important commands explicitly

if ! mysqldump --single-transaction shop > "$tmp/shop.sql"; then
  echo "dump failed" >&2
  exit 1
fi

cd /srv/app || exit 1

A die helper keeps this readable:

die() { echo "ERROR: $*" >&2; exit 1; }
cd /srv/app || die "cannot cd to /srv/app"
[[ -s "$tmp/shop.sql" ]] || die "dump is empty"

trap: clean up and report

Run cleanup however the script ends:

tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT

Report where a failure happened:

trap 'echo "error on line $LINENO: $BASH_COMMAND (exit $?)" >&2' ERR

Combine them with a notification for cron jobs:

on_error() {
  local code=$? line=$1
  echo "backup failed at line $line (exit $code)" >&2
  curl -fsS -m 10 --data "backup failed on $(hostname)" https://ntfy.sh/my-alerts > /dev/null || true
}
trap 'on_error $LINENO' ERR

Use set -E (errtrace) so the ERR trap also fires inside functions.

Retries for flaky steps

Network calls fail sometimes. Retry with a delay:

retry() {
  local n=1 max=5 delay=3
  until "$@"; do
    (( n >= max )) && { echo "failed after $n attempts: $*" >&2; return 1; }
    echo "attempt $n failed, retrying in ${delay}s" >&2
    sleep "$delay"; (( n++ )); (( delay *= 2 ))
  done
}

retry curl -fsS -o release.tar.gz "$URL"

curl -f matters: without it, curl exits 0 on HTTP 404 and 500 responses.

Atomic results

Write to a temporary file and move it into place only when everything has succeeded. Readers then never see half-written files:

mysqldump shop | gzip > "$dest/shop.sql.gz.tmp"
mv "$dest/shop.sql.gz.tmp" "$dest/shop.sql.gz"

Prune old backups after the new one is verified, never before.

Prevent overlapping runs

A cron job that takes longer than its interval can start twice. Use flock:

exec 9> /run/lock/backup.lock
flock -n 9 || { echo "already running" >&2; exit 1; }

Or in the crontab line: flock -n /run/lock/backup.lock /usr/local/bin/backup.sh. See crontab.

Logging

Send errors to stderr and keep a log:

exec > >(tee -a /var/log/backup.log) 2>&1

Under systemd, plain stdout and stderr go to the journal automatically. See tee.

ShellCheck

shellcheck script.sh (available via apt install shellcheck) finds unquoted variables, local masking errors, useless cd without || exit, and dozens of other bugs before they bite. Run it on every script. It’s also easy to add to CI.

A template

#!/usr/bin/env bash
set -Eeuo pipefail

die() { echo "ERROR: $*" >&2; exit 1; }
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
trap 'echo "failed at line $LINENO: $BASH_COMMAND" >&2' ERR

main() {
  [[ $# -ge 1 ]] || die "usage: $0 <db>"
  …
}

main "$@"

Bash
Scripting
Linux