When something goes wrong on a server, the answer is almost always in a log. Linux writes logs in two places: classic text files under /var/log, and the systemd journal. Knowing which log holds what saves a lot of time.

The most important log files

File Contains Distro
/var/log/syslog General system messages (services, cron, kernel) Debian/Ubuntu
/var/log/messages General system messages RHEL/AlmaLinux/CentOS
/var/log/auth.log Logins, SSH, sudo, authentication Debian/Ubuntu
/var/log/secure Same as auth.log RHEL family
/var/log/kern.log Kernel messages (also dmesg) Debian/Ubuntu
/var/log/dpkg.log, /var/log/apt/ Package installs and upgrades Debian/Ubuntu
/var/log/dnf.log Package operations RHEL family
/var/log/cron Cron (on RHEL; in syslog on Debian) RHEL family
/var/log/unattended-upgrades/ Automatic security updates Debian/Ubuntu
/var/log/fail2ban.log Bans and unbans If installed
/var/log/ufw.log Firewall blocks (if logging is on) If UFW is used
/var/log/wtmp, btmp, lastlog Binary login records (read with last, lastb) All

Application logs

Application Log location
Nginx /var/log/nginx/access.log, error.log. See Nginx logs
Apache /var/log/apache2/ (Debian) or /var/log/httpd/ (RHEL)
PHP-FPM /var/log/php8.3-fpm.log, plus pool slow logs
MySQL/MariaDB /var/log/mysql/error.log. See MySQL slow query log
PostgreSQL /var/log/postgresql/
Docker containers docker logs <container> (json-file or journald driver)
Mail (Postfix) /var/log/mail.log

The systemd journal

Most services also (or only) log to the journal, which journalctl reads:

journalctl -u nginx --since "1 hour ago"
journalctl -p err -b              # errors since this boot
journalctl -k                     # kernel messages
journalctl -f                     # follow everything live

See journalctl. On many systems, rsyslog copies journal messages into the text files above, so you can use either.

Reading logs efficiently

sudo tail -f /var/log/nginx/error.log           # follow live
sudo tail -n 100 /var/log/auth.log
sudo less +G /var/log/syslog                     # open at the end (Shift+F to follow)
sudo grep -i "error" /var/log/syslog | tail
sudo zgrep "Failed password" /var/log/auth.log*  # include rotated .gz files

See head and tail, less, and grep.

Rotation: why there are .1 and .gz files

logrotate renames logs on a schedule (syslog → syslog.1 → syslog.2.gz …), compresses old ones, and deletes the oldest. Configs live in /etc/logrotate.conf and /etc/logrotate.d/. See logrotate.

Logs and disk space

Logs are a common reason disks fill up: a debug log left on, an app logging every request, or a journal without limits.

sudo du -sh /var/log/* | sort -h | tail
journalctl --disk-usage
sudo journalctl --vacuum-size=500M

Limit the journal permanently with SystemMaxUse=500M in /etc/systemd/journald.conf. Never delete a log file that a process still has open: truncate it instead (sudo truncate -s 0 file.log), or the space isn’t freed. See disk full troubleshooting.

Central logging

For several servers, send logs to one place (rsyslog forwarding, Grafana Loki, ELK/OpenSearch, or a hosted service), so you can search across servers, and keep logs safe if a server is compromised.

Quick reference

Need Look at
Logins / SSH / sudo auth.log / secure
General system syslog / messages
Kernel kern.log, dmesg, journalctl -k
A service journalctl -u <svc>
Web errors /var/log/nginx/error.log
Package changes dpkg.log, apt/history.log, dnf.log

Related: Linux /tmp cleanup and check login history.

  • Linux
  • Logs
  • Troubleshooting