Every service on a modern Linux server, from Nginx and MySQL to your own Node or Python app, is described by a systemd unit file. Knowing how to read and write them lets you run your own apps reliably (auto-start, restart on crash, logs), and safely change the behaviour of packaged services.

Where unit files live

Path Contents Edit?
/usr/lib/systemd/system/ (or /lib/systemd/system/) Units installed by packages No, overwritten on upgrade
/etc/systemd/system/ Your units and overrides Yes
/etc/systemd/system/name.service.d/*.conf Drop-in overrides Yes
~/.config/systemd/user/ Per-user units Yes

Files in /etc take precedence over /usr/lib. See the effective unit, including drop-ins:

systemctl cat nginx

Anatomy of a service

# /etc/systemd/system/myapp.service
[Unit]
Description=My web application
Documentation=https://example.com/docs
After=network-online.target postgresql.service
Wants=network-online.target
Requires=postgresql.service

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/srv/myapp
EnvironmentFile=/srv/myapp/.env
ExecStart=/usr/bin/node server.js
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

[Unit]: description and ordering

  • After= — start after these units. It only sets order, not a requirement.
  • Wants= — also start these. A failure of the wanted unit is tolerated.
  • Requires= — also start these. If they fail or stop, this unit stops too.
  • network-online.target — use it (with both Wants and After) if your app needs working networking at startup, for example to reach a remote database.

[Service]: how to run it

Type — how systemd knows the service has started:

Type Use for
simple (default) The process runs in the foreground: Node, Gunicorn, most modern apps
exec Like simple, but startup fails if the binary can’t be executed
forking Old-style daemons that fork into the background (set PIDFile=)
oneshot Scripts that run and exit: backups, setup tasks. Often used with timers
notify Apps that tell systemd they’re ready (sd_notify): nginx, PostgreSQL…

Restart — on-failure restarts after crashes (non-zero exit, signal). always also restarts after a clean exit. Add RestartSec= to avoid tight loops. systemd stops retrying after too many restarts in a short time (StartLimitBurst/StartLimitIntervalSec in [Unit]).

User / Group — never run apps as root unless necessary. Create a system user first: sudo useradd -r -s /usr/sbin/nologin myapp.

Environment — Environment=NODE_ENV=production or EnvironmentFile=/path/.env (plain KEY=value lines, no export).

ExecStart — needs an absolute path, and no shell features (pipes, &&, ~) unless you wrap it: ExecStart=/bin/bash -c '…'.

[Install]: when to start

WantedBy=multi-user.target means “start at normal boot” once you enable it.

Load, enable and check

sudo systemctl daemon-reload            # after creating or changing unit files
sudo systemctl enable --now myapp
systemctl status myapp
journalctl -u myapp -f

See systemctl and journalctl.

Override packaged services: drop-ins

Never edit files in /usr/lib/systemd/system/. Create a drop-in instead:

sudo systemctl edit nginx

This opens an editor for /etc/systemd/system/nginx.service.d/override.conf. Add only what changes:

[Service]
LimitNOFILE=65535
Restart=on-failure

Saving reloads systemd automatically. Restart the service to apply.

To replace a list-type setting like ExecStart, clear it first:

[Service]
ExecStart=
ExecStart=/usr/sbin/nginx -g 'daemon off;' -c /etc/nginx/custom.conf

sudo systemctl edit --full name copies the whole unit to /etc for complete replacement. That’s less upgrade-friendly. systemctl revert name removes your overrides.

Sandboxing: cheap extra security

A few lines significantly limit what a compromised app can do:

[Service]
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/srv/myapp/uploads /var/log/myapp
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
  • ProtectSystem=strict makes the whole filesystem read-only except ReadWritePaths.
  • systemd-analyze security myapp scores the unit and suggests more options.

Add them gradually and test. Too strict and the app fails with permission errors that are visible in the journal.

Resource limits

[Service]
MemoryMax=1G
CPUQuota=150%
TasksMax=512
LimitNOFILE=65535

See OOM killer and ulimit.

Debugging units

systemd-analyze verify /etc/systemd/system/myapp.service    # syntax and reference checks
systemctl status myapp                                      # last lines of log, exit code
journalctl -u myapp -b --no-pager | tail -50
systemctl show myapp -p ExecStart -p User -p Environment    # effective values

Common errors:

  • code=exited, status=203/EXEC — ExecStart path wrong or not executable, or a bad shebang.
  • status=217/USER — the User= doesn’t exist.
  • status=200/CHDIR — WorkingDirectory is missing.
  • Works by hand, fails as a service — a different user, environment or PATH. Set them explicitly in the unit.

Templates: one unit, many instances

[email protected] with %i in it can run several instances:

sudo systemctl enable --now myworker@1 myworker@2

Quick reference

Task Command
Show effective unit systemctl cat name
Reload unit files systemctl daemon-reload
Override safely systemctl edit name
Remove overrides systemctl revert name
Verify systemd-analyze verify file
Security score systemd-analyze security name

Related: systemd for Node apps, systemd for Python apps and systemd timers.

  • Linux
  • systemd
  • DevOps