With one server, journalctl and tail -f are enough. With several servers, containers that come and go, and an incident at 3 a.m., you need all logs in one searchable place. Centralised log management also keeps logs safe if a server is compromised or lost.

Single-server basics: Linux log files explained and journalctl.

Why centralise

  • Search across servers and services at once (“all 5xx errors in the last hour”).
  • Correlate events by time, request ID or trace ID.
  • Keep logs after containers are recreated or servers are rebuilt.
  • Security: attackers often delete local logs. A remote copy survives.
  • Alerts on log patterns (failed logins, error spikes).

Structured logging first

Plain text logs are hard to query. Have applications log JSON, with consistent fields:

{"ts":"2026-11-05T10:15:22Z","level":"error","service":"billing-api","env":"prod","request_id":"a1b2","trace_id":"4bf9…","msg":"payment provider timeout","duration_ms":3012}

Then you can filter by service, level or request_id instead of guessing with regexes. See jq.

Choosing a stack

Stack Strengths Resource needs
Grafana Loki (+ Alloy/Promtail) Indexes only labels, so it’s cheap to store; great with Grafana and Prometheus Light; good on a small VPS
ELK / OpenSearch (+ Beats/Fluent Bit) Full-text search, powerful queries and dashboards Heavy: plan 8 GB+ RAM per node
Graylog (with OpenSearch + MongoDB) Friendly UI, streams, alerts, pipelines Medium–heavy
rsyslog/syslog-ng to a central server Simple, classic, files on disk Very light; limited search
Hosted services No operations work Cost per GB ingested

For small teams on VPS servers, Loki + Grafana is usually the most cost-effective choice. It pairs naturally with Prometheus metrics. See Grafana + Prometheus.

Shipping logs

Run a lightweight agent on each server:

  • Grafana Alloy (the successor of Promtail) for Loki;
  • Fluent Bit or Vector: fast, flexible, many outputs (Loki, OpenSearch, S3);
  • Filebeat for Elasticsearch/OpenSearch.

Sources to collect: the systemd journal, /var/log/* files (Nginx, auth), and Docker container logs. For Docker, either read the json-file logs from disk, or use the Loki Docker driver. See Docker logging drivers.

Keep consistent labels: host, service, env. In Loki, avoid high-cardinality labels (user IDs, request IDs) as labels. Keep those inside the log line.

Querying (Loki LogQL examples)

{service="nginx", env="prod"} |= " 502 "
{service="billing-api"} | json | level="error" | duration_ms > 2000
sum by (host) (count_over_time({service="sshd"} |= "Failed password" [5m]))

The last query turns logs into a metric you can alert on. See alerting best practices.

Retention and storage

  • Decide retention per type: application logs 14–30 days, security and audit logs longer (often 90+ days, depending on regulations).
  • Loki can store chunks on local disk or S3-compatible storage (MinIO), which makes long retention cheap. See MinIO.
  • Compress, and drop noisy debug logs at the agent.

Security and privacy

  • Send logs over TLS, or a private network/VPN. Don’t expose Loki or Elasticsearch publicly. See WireGuard.
  • Restrict who can read logs. They often contain IPs, emails and other personal data.
  • Don’t log secrets (passwords, tokens, full card numbers). Mask them in the app or the shipping pipeline.
  • Consider GDPR and similar rules for retention and access.

Quick reference

Need Choice
Cheap, simple, with Grafana Loki + Alloy
Full-text search at scale OpenSearch/ELK
UI-driven log management Graylog
Agents Alloy, Fluent Bit, Vector, Filebeat
Must-haves JSON logs, labels, retention, TLS

Related: observability basics and OpenTelemetry.

  • Logs
  • Monitoring
  • DevOps