Outages happen to everyone. What separates a 10-minute blip from a 4-hour disaster is usually preparation: knowing who does what, where to look, and what to try first. A lightweight incident process plus a handful of runbooks gives a small team most of the benefit, without bureaucracy.

Severity levels

Severity Example Response
SEV1 Site or checkout down for everyone; data loss; security breach All hands, immediately; status page update
SEV2 Major feature broken, or severe slowness for many users On-call responds now; notify the team
SEV3 Minor feature broken, workaround exists Fix during working hours

Roles (even if it’s just two people)

  • Incident lead: coordinates, decides, keeps focus on mitigation.
  • Investigator(s): dig into logs, metrics and changes.
  • Communicator: updates the status page, customers and stakeholders, so investigators can work.

In a team of one or two, roles combine, but still separate “fixing” time from “communicating” time.

The response process

  1. Acknowledge the alert, and open an incident channel or document (timestamps matter).
  2. Assess impact: who is affected, and how badly? Set the severity.
  3. Communicate early: “We’re investigating issues with checkout” on the status page within 10–15 minutes.
  4. Mitigate first: roll back the last deploy, fail over, scale up, block abusive traffic, or disable a feature flag. Restore service before finding the root cause. See deployment strategies and feature flags.
  5. Investigate systematically. See Linux troubleshooting methodology.
  6. Resolve and verify with real user-facing checks.
  7. Communicate resolution, then schedule a postmortem.

Runbook template

Keep runbooks short, in a wiki or the repository, linked from alerts:

# Runbook: Website returns 502 Bad Gateway

**Alert:** HighErrorRate / Uptime check failing
**Impact:** Users can't load pages

## Check
1. `systemctl status php8.3-fpm nginx`
2. `sudo tail -n 50 /var/log/nginx/error.log`
3. `free -h` and `df -h`

## Mitigate
- PHP-FPM down: `sudo systemctl restart php8.3-fpm`
- "pm.max_children reached": raise temporarily, find slow requests
- Disk full: clear logs/cache (see Disk Full runbook)
- After a deploy: roll back to previous release tag

## Escalate
- Still down after 15 min: call <name>, check provider status / VNC console

Runbooks worth writing first

Runbook Starting points
Site down / 502 / 504 Nginx 502, Nginx 504
Disk full Disk full troubleshooting
Server unreachable VPS VNC console, provider status
Database down or slow MySQL slow query log
High load / slow site VPS performance troubleshooting
Certificate expired Let’s Encrypt
Suspected compromise Detect a compromised server
DDoS / traffic spike VPS DDoS protection
Restore from backup Linux disaster recovery

Status page

A public status page (for example Uptime Kuma’s status pages, or Cachet, or a hosted service) reduces support load during incidents. Host it separately from the main infrastructure, so it stays up when your servers don’t. See Uptime Kuma.

Postmortems

Within a few days, write up:

  • summary and impact (duration, users affected);
  • timeline (detection, mitigation, resolution);
  • root causes and contributing factors;
  • what went well, and what didn’t;
  • action items with owners and dates.

Keep them blameless. See SRE basics.

Prepare before you need it

  • Make sure more than one person can access servers, DNS, registrar, and the hosting client area (with 2FA and a shared password manager).
  • Keep runbooks, credentials and contact lists outside the infrastructure they describe.
  • Run a game day: simulate an outage on staging, and walk through the process.

Quick reference

Step Goal
Acknowledge Someone owns it
Assess Severity and impact
Communicate Status page within minutes
Mitigate Rollback, failover, flag off
Investigate Root cause, systematically
Postmortem Learn, assign actions

Related: alerting best practices and observability basics.

  • SRE
  • DevOps
  • Reliability