Hardening reduces the chance of a compromise. Monitoring shortens the time until you notice one, and that time decides how much damage is done. You don’t need a security operations centre to monitor a few servers. You need to collect the right signals, alert on the few that matter, and look at them.

The signals that matter

1. Authentication

  • Successful SSH logins, especially from new IPs. See SSH login alerts.
  • sudo use and root shells. See sudo logging.
  • New users, new SSH keys, and changes to the sudo, wheel or docker groups.
  • Spikes in failed logins (useful context, though brute force alone isn’t urgent with key-only SSH).

2. Changes to the system

  • File integrity: system binaries, configs, web application code. See AIDE.
  • New cron jobs, systemd units, and authorized_keys changes. See auditd.
  • Packages installed or removed outside maintenance windows.

3. Processes and resources

  • CPU pegged without traffic: often a crypto-miner. See high CPU usage.
  • Unknown processes, or processes running from /tmp or /dev/shm.
  • Sudden disk growth: dumps, exfiltration staging, log floods.

4. Network

  • Unexpected outbound connections: mining pools, IRC, mass SMTP, scanning other hosts.
  • New listening ports (ss -tlnp compared with a baseline).
  • Traffic spikes, inbound and outbound.
  • Provider abuse notifications. Treat them as a high-priority alert.

5. Web applications

  • Requests to known exploit paths, and web shell names.
  • New admin users in the CMS.
  • Changed or new PHP files in upload directories.
  • Error bursts (500s) after suspicious requests.

A practical stack by size

One or a few servers

  • Logs: journald, plus rsyslog forwarding to a second machine or a log service. See rsyslog.
  • Alerts: PAM login notifications, Fail2Ban or CrowdSec, AIDE daily reports, resource alerts from Netdata. See Netdata.
  • Audits: monthly Lynis runs. See Lynis.
  • External checks: uptime and certificate monitoring (Uptime Kuma), and periodic nmap scans of your public IPs from outside. See Uptime Kuma and nmap.

Growing fleets

  • Central logs: Grafana Loki, Graylog or OpenSearch, with alert rules on auth, sudo and audit events.
  • Host-based IDS / SIEM: Wazuh (open source), with file integrity, log analysis, vulnerability detection and a dashboard. See Wazuh.
  • Network IDS: Suricata on gateways or high-value hosts. See Suricata.
  • Metrics: Prometheus and Grafana, with alerts on anomalies. See Prometheus and Grafana.

Make alerts useful

The most common failure isn’t missing data, it’s ignored alerts:

  • Alert only on things that need action: a new SSH key, a root login from an unknown IP, a changed system binary, outbound SMTP spikes.
  • Route alerts to a channel people actually watch (Telegram, Slack, email with filters), with severity levels.
  • Suppress known-good events (your own deploys, backup jobs) so the real ones stand out.
  • Review noisy alerts monthly. Fix the source, or tune the rule.

Protect the monitoring itself

An attacker with root can disable local agents and wipe logs. Mitigations:

  • Ship logs off the server in real time. What’s already sent can’t be erased.
  • Alert on silence: if a server stops sending logs or heartbeats, that’s an alert too.
  • Keep monitoring credentials separate from server credentials.

Daily and weekly habits

Daily (automated summaries): logins, sudo, Fail2Ban bans, AIDE changes, resource anomalies.

Weekly: pending security updates and reboots (patch management), backup success, certificate expiry.

Monthly: Lynis audit, an external port scan, a review of users, keys and sudo rules (least privilege).

When an alert fires

Have a short runbook ready: who checks, how to isolate the server (firewall rules, provider panel), where the backups are, and the rebuild procedure. See detect a compromised server.

Related: Linux server hardening and brute-force protection.

  • Security
  • Monitoring
  • Linux