Site Reliability Engineering (SRE) started at Google, but its core ideas work for a two-person team running a SaaS on a few VPS servers: decide how reliable the service needs to be, measure it, and use that number to balance new features against stability work. You don’t need a dedicated SRE team to benefit.
SLIs, SLOs and SLAs
| Term | Meaning | Example |
|---|---|---|
| SLI (indicator) | A measured aspect of service quality | % of HTTP requests that succeed; % served in < 500 ms |
| SLO (objective) | The target for an SLI over a period | 99.9% of requests succeed over 30 days |
| SLA (agreement) | A contractual promise, usually with penalties | 99.5% monthly availability, or service credits |
Set SLOs stricter than any SLA you promise, so you notice problems before they cost money. See VPS uptime and SLAs.
Choosing good SLIs
Measure what users experience:
- Availability: successful requests / total requests (from load balancer or app metrics, not “is the server pingable”).
- Latency: share of requests faster than a threshold (for example 95% under 400 ms).
- Correctness or freshness for data pipelines (jobs completed on time).
# availability SLI over 30 days (Prometheus)
sum(rate(http_requests_total{status!~"5.."}[30d])) / sum(rate(http_requests_total[30d]))See observability basics.
Error budgets
If your SLO is 99.9% over 30 days, your error budget is 0.1%: about 43 minutes of full downtime, or the equivalent in failed requests.
- Budget left: ship features, run experiments, deploy freely.
- Budget burned: slow down risky changes, and focus on reliability fixes until it recovers.
The budget turns “should we spend time on stability?” into a data-driven decision, and stops endless perfectionism, because 100% reliability is neither possible nor worth the cost.
Burn-rate alerts page you when the budget is being consumed fast (for example, at a rate that would exhaust it in a few days), which is better than static thresholds. See alerting best practices.
Toil
Toil is manual, repetitive, automatable work that scales with the service: restarting a stuck worker by hand, renewing certificates manually, copying files for deploys. SRE aims to keep toil under roughly half of operations time, and to automate the rest:
- certificates → Certbot or cert-manager. See Let’s Encrypt;
- deploys → CI/CD. See CI/CD pipeline explained;
- server setup → Ansible. See Linux configuration management;
- restarts → systemd
Restart=, health checks, and fixing the root cause.
Incidents and blameless postmortems
When something breaks:
- Restore service first (mitigate), then investigate.
- Write a short postmortem: timeline, impact, root causes, what went well, and action items with owners.
- Keep it blameless: focus on systems and processes, not people. People hide problems when they fear blame.
See incident response runbooks.
A lightweight starting plan
- Pick one SLO per user-facing service (availability), and maybe one for latency.
- Measure it from real traffic, and show it on a dashboard.
- Alert on fast budget burn, not on every blip.
- After each incident, write a one-page postmortem.
- Every month, automate one piece of toil.
Reliability architecture basics
SLOs tell you how much reliability you need. Architecture provides it: backups, redundancy, health checks, graceful degradation, and capacity headroom. See VPS high availability and Linux disaster recovery plan.
Quick reference
| Concept | One-liner |
|---|---|
| SLI | What you measure |
| SLO | The target |
| Error budget | 100% − SLO; spend it on change |
| Burn rate | How fast the budget is being used |
| Toil | Manual repetitive work; automate it |
| Postmortem | Blameless learning with action items |
Related: DevOps for small teams and deployment strategies.