You start a long task over SSH, such as a big download, a database import or a migration, and then your connection drops. Normally the shell sends SIGHUP to its jobs when it exits, and they die. This guide covers the shell’s job control, nohup and disown, and the better long-term options: tmux and systemd.

Background jobs with &

./long-task.sh &
# [1] 24871

The shell prints the job number [1] and the process ID, and gives you the prompt back. Output still goes to your terminal, so redirect it:

./long-task.sh > task.log 2>&1 &

Managing jobs: jobs, fg, bg

jobs            # list jobs in this shell
fg %1           # bring job 1 to the foreground
bg %1           # resume a stopped job in the background
kill %1         # terminate job 1

If you started something in the foreground and want it back in the background:

  1. Press Ctrl+Z to suspend it. It shows [1]+ Stopped.
  2. Type bg to continue it in the background.

Background jobs belong to your shell. Close the shell and they normally receive SIGHUP.

nohup: ignore the hangup

nohup ./long-task.sh > task.log 2>&1 &

nohup runs the command with SIGHUP ignored, so it keeps running after you log out. If you don’t redirect, output goes to nohup.out in the current directory (or ~/nohup.out).

Check on it later:

tail -f task.log
pgrep -af long-task

Typical uses:

nohup mysql shop < dump.sql > import.log 2>&1 &
nohup wget -c https://example.com/big.iso > /dev/null 2>&1 &
nohup python3 train.py > train.log 2>&1 &

disown: rescue a job you’ve already started

Forgot nohup? Detach a running job from the shell:

# job is running in the foreground
Ctrl+Z
bg
disown -h %1     # keep the job, but don't send it SIGHUP
# or
disown %1        # remove it from the job table entirely

It keeps running after logout. Its output still points at the old terminal, though. Writes to a closed terminal can fail, and some programs exit when that happens. For anything important, redirect output from the start.

setsid: fully detach

setsid ./long-task.sh > task.log 2>&1 < /dev/null &

The process gets its own session, with no controlling terminal at all. It’s useful for daemons started by hand.

Better: tmux or screen

For interactive or long-running work over SSH, a terminal multiplexer is better than nohup:

tmux new -s import
mysql shop < dump.sql
# detach: Ctrl+B then D
# later, after reconnecting:
tmux attach -t import

You see live output, you can interact, and nothing is lost if the connection drops. See tmux and screen.

Best for services: systemd

If something should run permanently, such as a web app, a bot or a worker, don’t use nohup. Use a systemd service. It starts on boot, restarts on crash, collects logs and runs as a dedicated user:

[Service]
ExecStart=/usr/bin/node /srv/app/server.js
Restart=always
User=app

See systemd services for Node.js and systemctl.

For a one-off task that should survive logout, systemd-run is a neat option:

sudo systemd-run --unit=reindex --property=User=app /srv/app/bin/reindex
journalctl -u reindex -f

It runs as a transient service with logs in the journal.

Which to use

Situation Use
Quick task, don’t care about output cmd &
One-off long task, check the log later nohup cmd > log 2>&1 &
Already running, forgot nohup Ctrl+Z, bg, disown -h
Interactive or want to watch progress tmux / screen
Runs permanently, should restart systemd service
One-off as a managed unit systemd-run

Gotchas

  • nohup doesn’t make a process immune to kill or to the OOM killer. It only ignores SIGHUP.
  • Some systemd setups (KillUserProcesses=yes in logind) kill all your processes at logout, even nohup’d ones. Use loginctl enable-linger user or a systemd unit.
  • sudo nohup cmd & works, but nohup sudo cmd & may stop to ask for a password in the background. Authenticate first with sudo -v.

Related: ps and kill.

  • Linux
  • Bash
  • Command Line