Good alerting wakes you up for problems users notice, and stays quiet otherwise. Bad alerting does the opposite: dozens of noisy notifications that everyone learns to ignore, until the one that matters gets ignored too. Here’s how to design alerts that are worth acting on.

Principles

  1. Every alert must be actionable. If nobody needs to do anything, it isn’t an alert. Make it a dashboard or a report.
  2. Alert on symptoms, not causes. “Checkout error rate above 2%” matters. “CPU at 85%” might not. Use cause metrics on dashboards and for diagnosis.
  3. Use durations. Alert when a condition persists (for example 5 minutes), not on a single spike.
  4. Route by severity. Page someone only for urgent, user-impacting problems. Everything else goes to a ticket or chat.
  5. Include context. What’s wrong, how bad it is, where to look (dashboard link), and what to do (runbook link).

Severity levels

Severity Meaning Delivery
Critical / page Users affected now, or will be soon Phone push / SMS / call, any hour
Warning Needs attention within hours (disk at 85% and filling) Chat channel, email
Info FYI, no action required Dashboard or daily digest, not an alert

Alerts every VPS setup needs

Alert Why
Website / API down (external check) The most important symptom. See Uptime Kuma
High error rate (5xx) Users are failing
Latency p95 above target Users are waiting
Disk will fill within N hours Prevents outages. See VPS storage planning
SSL certificate expires in < 14 days Easy to prevent, embarrassing to miss
Backup job failed or didn’t run Silent backup failures are disasters waiting to happen
Server unreachable / exporter down You’re blind otherwise
Memory exhaustion / OOM kills Crashes. See Linux OOM killer

Prometheus alert rules (examples)

groups:
  - name: basics
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 2m
        labels: { severity: critical }
        annotations:
          summary: "{{ $labels.instance }} is down"
          runbook: "https://wiki.example.com/runbooks/instance-down"

      - alert: DiskWillFillIn6Hours
        expr: predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[1h], 6*3600) < 0
        for: 15m
        labels: { severity: warning }
        annotations:
          summary: "Disk on {{ $labels.instance }} {{ $labels.mountpoint }} will fill within 6h"

      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
            / sum(rate(http_requests_total[5m])) > 0.02
        for: 5m
        labels: { severity: critical }

predict_linear alerts on the trend (“will be full soon”) instead of a fixed threshold. See Prometheus node exporter and Grafana + Prometheus.

Routing with Alertmanager

route:
  receiver: chat
  group_by: [alertname, instance]
  group_wait: 30s
  repeat_interval: 4h
  routes:
    - matchers: [severity="critical"]
      receiver: pager
receivers:
  - name: chat
    telegram_configs: [{ bot_token: "…", chat_id: 123456 }]
  - name: pager
    webhook_configs: [{ url: "https://ntfy.example.com/oncall" }]

Grouping combines related alerts into one notification. Inhibition silences dependent alerts (no “website down” pages when the whole server is down), and silences mute alerts during planned maintenance.

Delivery options: email, Slack/Discord/Telegram, push notifications (ntfy, Pushover), and on-call tools (PagerDuty, Grafana OnCall).

Fighting alert fatigue

  • Review alerts monthly. Delete or downgrade any alert that fired without action.
  • Tune thresholds and for: durations using real data.
  • Never “temporarily” ignore a flapping alert for weeks. Fix it, or remove it.
  • Track how many pages per week each person gets. Too many leads to burnout, and missed incidents.

Test your alerts

Alerts that never fire might be broken. Periodically test: stop a test service, fill a test disk, expire a test certificate. Make sure the notification arrives where you expect.

Quick reference

Rule Practice
Actionable Every alert has a clear response (runbook)
Symptoms Errors, latency, availability first
Durations for: 5m to avoid flapping
Severity Page only for critical
Trends predict_linear for disks
Maintenance Silences, not disabled alerts

Related: incident response runbooks and observability basics.

  • Monitoring
  • DevOps
  • SRE