SLOs Turn Reliability Into a Product Decision

At 2:13 a.m., the API was technically up. Health checks passed, pods were running, and the load balancer was returning traffic. Customers were still watching requests time out.

The incident bridge spent twenty minutes arguing about whether the service was actually unavailable. Operations had an uptime graph. Developers had application logs. Product had support tickets. Everyone brought data, but nobody had an agreed definition of acceptable service.

That is the problem service level objectives solve. Not monitoring. Not dashboards. They turn reliability from a vague engineering aspiration into a product decision with numbers, ownership, and consequences.

Uptime is usually the wrong argument

An SLO sets a target for a service level indicator, or SLI. The SLI measures something users experience. The SLO states how often that experience must meet an agreed threshold over a defined period.

For an HTTP API, useful indicators might include successful request ratio and the proportion of requests completing within 500 milliseconds. CPU usage is not an SLI. Pod availability is not an SLI. Those measurements help diagnose a problem, but customers do not care whether a pod is healthy while checkout is failing.

Even a statement such as “99.9 percent available” is incomplete. Available to whom? Which endpoints count? Do client errors count? What about background jobs, planned maintenance, retries, or requests rejected by rate limits?

A usable objective needs boundaries. For example:

• 99.9 percent of eligible checkout requests return a non-5xx response over a rolling 30-day window.
• 99 percent of eligible checkout requests complete within 750 milliseconds over the same window.
• Synthetic traffic, internal health checks, and known client errors are excluded.

Those details are where the real work lives. Teams often rush past them because choosing a percentage feels easier than agreeing what the service actually promises.

The budget is the useful part

A 99.9 percent success objective permits 0.1 percent failure. That allowance is the error budget.

For a purely time-based service, 99.9 percent over 30 days allows 43 minutes and 12 seconds outside the objective. Request-based budgets are often more representative. If a service handles 10 million eligible requests during the window, its budget allows 10,000 bad requests.

This is not permission to cause incidents. It is a control mechanism for balancing reliability work against feature delivery. When the service has most of its budget, teams can accept normal deployment risk. When it burns the budget too quickly, release policy should change.

A practical policy might allow routine releases while more than half the budget remains. Below that threshold, risky migrations and large changes need additional review. Once the budget is exhausted, reliability fixes take priority over feature work until the service returns to policy.

The exact thresholds can vary, but the response cannot be optional. An error budget that changes nothing is just another dashboard tile.

Measure the user path, not the machinery

Prometheus can calculate a request success ratio directly from application metrics. The basic query is simple:

sum(rate(http_requests_total{service="checkout",status!~"5.."}[5m]))
/
sum(rate(http_requests_total{service="checkout"}[5m]))

Production queries usually need more care. Exclude health probes and endpoints outside the SLO. Decide how cancelled requests are classified. Check that every replica emits labels consistently. Avoid grouping by dimensions such as customer ID that can produce unmanageable metric cardinality.

Latency objectives should use histogram buckets rather than averages. An average can look healthy while a smaller but significant group of customers waits several seconds.

sum(rate(http_request_duration_seconds_bucket{
 service="checkout",
 le="0.75"
}[5m]))
/
sum(rate(http_request_duration_seconds_count{
 service="checkout"
}[5m]))

This query asks a product question: what proportion of checkout requests finished within 750 milliseconds? That is more useful than asking whether average latency remained below some line selected years ago.

Tools such as Prometheus, Grafana, OpenSLO, Sloth, Pyrra, and OpenTelemetry can support this model. The technology is not the hard part. The hard part is agreeing on service boundaries and instrumenting them consistently.

Stop paging on symptoms that do not threaten the objective

Static alerts tend to produce noise. A five-minute error spike may page somebody even though retries hide the impact and the monthly objective remains healthy. A slow, persistent failure can do the opposite, quietly consuming the budget without crossing a dramatic threshold.

Burn-rate alerts measure how quickly the service is spending its error budget. For a 99.9 percent objective, the allowed bad-event ratio is 0.001. If the current bad-event ratio is 0.0144, the service is burning budget at 14.4 times the sustainable rate.

High burn across both short and longer windows deserves a page because the budget will disappear quickly if the condition continues. Lower burn over several hours may warrant a ticket and investigation during working hours. Requiring both windows helps filter brief spikes without ignoring sustained damage.

This gives the person on call a better reason to wake up: customers are being affected badly enough to threaten an agreed reliability target. The alert is tied to policy, not merely to a metric behaving unusually.

Architecture now has a price tag

SLOs expose uncomfortable architecture decisions. A team asking for 99.99 percent availability may need isolation between failure domains, safer database migrations, tested failover, capacity headroom, and fewer synchronous dependencies. In cloud systems, those choices cost money and operational attention.

That makes SLOs relevant to FinOps. Cost discussions become clearer when teams can connect infrastructure spending to a reliability target. Paying for another region may be justified for one customer-facing transaction and wasteful for an internal reporting job. The target should follow the business impact, not the ambitions of whoever designed the platform.

The same clarity improves developer experience. Engineers should be able to see an SLO beside deployment and incident data, understand which change consumed budget, and know what release rules apply. If using the SLO requires navigating six dashboards and asking the platform team to interpret them, adoption will stall.

Reliability needs an operating routine

Review SLOs with the same discipline used for product metrics. Service owners should inspect budget consumption, major contributors, measurement gaps, and upcoming changes. Product representatives need to be present when targets or policies change because engineering cannot decide the acceptable customer impact alone.

Do not silently reset a budget after an incident. Do not exclude an outage because it makes the report look bad. Fixing bad instrumentation is reasonable. Editing the definition after seeing the result destroys trust.

Targets also need occasional correction. An objective that is always exceeded by a wide margin may be too loose, or the architecture may be more expensive than necessary. One that is permanently missed without any operational response is fiction.

If you only do one thing this week

Choose one customer-facing service and write a single request-based SLO in plain language. Name the eligible requests, success condition, window, target, owner, and action taken when the budget is exhausted.

Then calculate it from production telemetry, even if the first result is messy. The gaps will show you where reliability is currently based on assumption rather than evidence. That is useful information. Start there.

TecLeads helps engineering teams ship this kind of thing faster and more safely. If you'd like a second pair of eyes on your setup, book a 30-minute call or explore what we do.