rsyslog is the syslog daemon on most Debian, Ubuntu and RHEL systems. It receives log messages from the kernel, services and journald, and writes them to files like /var/log/syslog and /var/log/auth.log. It can also forward them to a central log server. Knowing how it routes messages helps you find logs, split noisy ones into their own files, and keep copies off the server.

rsyslog and journald

On systemd systems, journald collects everything first and passes messages on to rsyslog, which writes the classic text files.

  • journalctl reads the journal (binary, structured). See journalctl.
  • rsyslog writes /var/log/* text files, which are easy to grep, tail and ship elsewhere.

Debian 12+ no longer installs rsyslog by default, and relies on the journal only. Install it if you want text logs or remote forwarding:

sudo apt install rsyslog

Where logs go

File Contents
/var/log/syslog (Debian/Ubuntu) / /var/log/messages (RHEL) General system messages
/var/log/auth.log / /var/log/secure Logins, SSH, sudo
/var/log/kern.log Kernel messages
/var/log/mail.log / /var/log/maillog Mail server
/var/log/cron.log (if enabled) Cron jobs

Configuration lives in /etc/rsyslog.conf and /etc/rsyslog.d/*.conf. The defaults on Ubuntu are in /etc/rsyslog.d/50-default.conf.

Facilities and severities

Classic rules look like facility.severity destination:

auth,authpriv.*        /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
kern.*                 -/var/log/kern.log
mail.*                 -/var/log/mail.log
  • Facility is the source category: auth, authpriv, cron, daemon, kern, mail, user, and local0–local7 for your own use.
  • Severity, from most to least severe: emerg, alert, crit, err, warning, notice, info, debug. mail.err means err and above.
  • none excludes a facility. A leading - on the file means asynchronous writes, which are faster.

Send a program’s logs to its own file

Use a property-based filter on the program name. For example, put a noisy service in its own file and keep it out of syslog:

# /etc/rsyslog.d/30-myapp.conf
if $programname == 'myapp' then {
    action(type="omfile" file="/var/log/myapp.log")
    stop
}

stop prevents the message from also landing in /var/log/syslog.

Other useful filters:

:msg, contains, "healthcheck" stop                         # drop noise
if $syslogseverity <= 3 then /var/log/errors.log           # err and worse from everything

Apply and test:

sudo rsyslogd -N1                 # check config syntax
sudo systemctl restart rsyslog
logger -t myapp "test message"    # send a test line
tail /var/log/myapp.log

logger is handy in shell scripts too: logger -t backup "backup finished".

Add the new file to logrotate, or it will grow forever. See logrotate.

Enable a cron log (Debian/Ubuntu)

Uncomment this in 50-default.conf:

cron.*    /var/log/cron.log

Forward logs to a central server

Keeping a copy of logs on another machine helps with troubleshooting across servers, and with security. An attacker who gets root can wipe local logs, but not ones that were already shipped.

On the log server:

# /etc/rsyslog.d/10-receive.conf
module(load="imtcp")
input(type="imtcp" port="514")

template(name="PerHost" type="string" string="/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log")
if $fromhost-ip != '127.0.0.1' then {
    action(type="omfile" dynaFile="PerHost")
    stop
}

Allow port 514/tcp only from your servers’ IPs. See UFW.

On each client:

# /etc/rsyslog.d/90-forward.conf
*.* action(type="omfwd" target="10.0.0.20" port="514" protocol="tcp"
           queue.type="LinkedList" queue.filename="fwd" queue.saveOnShutdown="on"
           action.resumeRetryCount="-1")

The disk-assisted queue buffers messages while the log server is unreachable, so nothing is lost.

Over the public internet, encrypt the traffic. rsyslog supports TLS (gtls driver), or you can send through a WireGuard tunnel. See WireGuard.

rsyslog vs a log stack

For a handful of servers, central rsyslog plus grep works well. For search, dashboards and alerts across many servers, ship logs into a log stack: Grafana Loki with Promtail or Alloy, Graylog, or OpenSearch. rsyslog can feed most of them.

Troubleshooting

  • No logs in /var/log/syslog: is rsyslog installed and running (systemctl status rsyslog)? On Debian 12+ it isn’t installed by default.
  • Config changes ignored: run rsyslogd -N1 for syntax errors, and check the file order in rsyslog.d (files are processed alphabetically, and an earlier stop wins).
  • Permission denied writing files: on Ubuntu, rsyslog runs as the syslog user. New log directories must be writable by it.
  • Disk filling: rate-limit noisy sources, drop noise with filters, and make sure logrotate covers every file.

Quick reference

Task Command / config
Check syntax rsyslogd -N1
Restart systemctl restart rsyslog
Test message logger -t tag "text"
Own file per program if $programname == 'x' then { action(…) stop }
Forward over TCP action(type="omfwd" target="IP" port="514" protocol="tcp")
Receive module(load="imtcp") input(type="imtcp" port="514")

Related: Nginx logs and Fail2Ban.

  • Linux
  • Logs
  • Configuration