Monitoring tells you that something is wrong (the site is down, CPU is at 100%). Observability helps you understand why: which endpoint got slow, which deploy caused it, which database query is to blame. It comes from three kinds of telemetry (metrics, logs and traces) collected consistently, and connected to each other.

The three pillars

Signal What it is Answers Tools
Metrics Numbers over time (counters, gauges, histograms) “Is error rate rising? How fast are requests?” Prometheus, VictoriaMetrics
Logs Timestamped event records “What exactly happened in this request?” Loki, OpenSearch/ELK
Traces The path of one request across services, with timings “Where did the 3 seconds go?” Tempo, Jaeger

Plus profiles (where CPU and memory are spent in code), increasingly seen as a fourth signal. See perf profiling.

What to measure first: golden signals

For every user-facing service, track:

  1. Latency: how long requests take (p50, p95, p99, not just the average).
  2. Traffic: requests per second.
  3. Errors: rate of failed requests (5xx, exceptions).
  4. Saturation: how full the resources are (CPU, memory, disk, connection pools, queues).

For servers, the USE method (utilisation, saturation, errors) covers each resource. See Linux troubleshooting methodology.

A practical self-hosted stack

Need Tool
Server metrics node_exporter → Prometheus. See Prometheus node exporter
App metrics Prometheus client libraries, or OpenTelemetry
Dashboards Grafana. See Grafana + Prometheus
Logs Grafana Loki + Alloy/Promtail (or OpenSearch)
Traces Grafana Tempo or Jaeger, fed by OpenTelemetry. See OpenTelemetry
Uptime from outside Uptime Kuma. See Uptime Kuma
Simple all-in-one Netdata. See Netdata

Run the stack on a separate VPS from the services it watches, so it keeps working (and alerting) when they fail. A 2–4 GB VPS handles metrics and logs for a handful of servers. Size storage for your retention period. See VPS storage planning.

Connect the signals

Observability shines when you can jump between signals:

  • a dashboard shows a latency spike at 14:05;
  • exemplars or trace IDs link it to slow traces;
  • the trace shows a slow database call, and its logs (with the same trace ID) show the query.

To make that possible:

  • log in structured JSON, and include trace_id, request_id and user or tenant IDs;
  • use consistent labels (service, env, instance) across metrics, logs and traces;
  • instrument with OpenTelemetry, which propagates trace context automatically.

Start small

  1. External uptime checks for every public endpoint.
  2. node_exporter + Prometheus + Grafana for every server.
  3. Centralised logs with retention and search. See log management.
  4. App metrics for the golden signals (most frameworks have middleware for this).
  5. Tracing for services that call other services or slow dependencies.
  6. Alerts on symptoms users feel. See alerting best practices.

Costs and pitfalls

  • Cardinality: labels with unbounded values (user IDs, full URLs) explode metric storage. Keep those in logs and traces.
  • Log volume: debug logs everywhere get expensive fast. Sample, and set retention.
  • Dashboards nobody reads: design dashboards around questions you’ll actually ask during incidents.

Quick reference

Signal Best for Self-hosted tool
Metrics Trends, alerts Prometheus + Grafana
Logs Details, debugging Loki / OpenSearch
Traces Latency breakdown across services Tempo / Jaeger
Uptime User’s view Uptime Kuma
Standard instrumentation All of the above OpenTelemetry

Related: SRE basics and incident response runbooks.

  • Monitoring
  • DevOps
  • Observability