Skip to content

SLIs, SLOs, SLAs & Error Budgets

Modern reliability and performance engineering distinguishes sharply between what you measure (SLI), what your engineering team targets internally (SLO), and what your business legally promises customers (SLA). Conflating the three leads either to alert fatigue or costly contract breaches.


1. The Reliability Hierarchy: SLI → SLO → SLA

flowchart LR
    SLI["SLI (Indicator)<br/>Quantitative measurement<br/>e.g., P95 of valid 2xx responses = 182ms"] --> SLO["SLO (Objective)<br/>Internal engineering target<br/>e.g., P95 < 250ms over 28 days"]
    SLO --> EB["Error Budget<br/>100% - SLO<br/>Room for releases, spikes & experiments"]
    SLO --> SLA["SLA (Agreement)<br/>External contract + financial credits<br/>e.g., P95 < 500ms, Availability >= 99.9%"]
Layer Full Name Owner Example Consequence if Breached
SLI Service Level Indicator Engineering / SRE % of HTTP requests completing < 250ms with non-5xx status None by itself—it is the speedometer
SLO Service Level Objective Product + Engineering 99.5% of requests in 28-day window satisfy the latency SLI Error budget depleted; freeze risky feature rollouts, prioritize performance/reliability work
SLA Service Level Agreement Legal / Sales + Leadership Monthly uptime >= 99.9% & P95 < 500ms Financial refunds / service credits, reputational damage, contract termination

Golden Rule: Set Internal SLOs Stricter Than External SLAs

If your customer contract (SLA) promises P95 < 500 ms and 99.9% availability, set your internal SLO (and Gatling CI assertions!) at P95 < 300 ms and 99.95%. That margin gives your team early warning to fix regressions before customers or contracts are impacted.


2. Availability "Nines" & Error Budgets

An Error Budget is 100% - SLO. It quantifies the exact volume of failed/slow requests or downtime permissible within a rolling window:

Target ("Nines") Error Budget % Allowable Downtime / Month Allowable Downtime / Year Failed Requests per 10M Requests
99% ("Two nines") 1.000% 7 hr 18 min 3.65 days 100,000
99.5% 0.500% 3 hr 39 min 1.83 days 50,000
99.9% ("Three nines") 0.100% 43 min 50 sec 8 hr 46 min 10,000
99.95% 0.050% 21 min 55 sec 4 hr 23 min 5,000
99.99% ("Four nines") 0.010% 4 min 23 sec 52 min 36 sec 1,000
99.999% ("Five nines") 0.001% 26.3 seconds 5 min 15 sec 100

Request-Based Error Budgets in Load Testing

Time-based uptime hides partial brownouts! Always express load-test SLOs as request-based budgets:

Good Request Ratio = ( Requests with Status 2xx/3xx AND Latency <= Threshold ) / Total Valid Requests

In Gatling Java DSL, encode both availability and percentile SLOs directly as build-breaking .assertions():

setUp(
    checkoutScenario.injectOpen(constantUsersPerSec(200).during(Duration.ofMinutes(15)))
)
.protocols(httpProtocol)
.assertions(
    global().responseTime().percentile(95.0).lt(300),     // Latency SLO (stricter than 500ms SLA)
    global().responseTime().percentile(99.0).lt(800),     // Tail Latency SLO
    global().successfulRequests().percent().gte(99.9),    // Error Budget <= 0.1%
    forAll().responseTime().percentile(99.0).lt(1200)     // Every individual endpoint < 1.2s
);

3. Composite Availability & Latency Budgeting in Microservices

Serial Dependency Availability Math

If a single POST /checkout call synchronously invokes 5 microservices in series (Gateway $\rightarrow$ Auth $\rightarrow$ Cart $\rightarrow$ Pricing $\rightarrow$ Payment), and each service individually achieves 99.9% availability:

Composite Availability = 0.999^5 = 0.995009 ≈ 99.5% (5× more downtime!)

To guarantee a 99.9% public API SLO across 5 synchronous hops, each internal downstream dependency must achieve $\ge 99.98\%$—or be decoupled asynchronously (e.g., via Kafka or cached fallbacks).

End-to-End Latency Budgeting

Allocate your public P95 latency budget explicitly across architectural tiers before load testing:

flowchart LR
    Total["Public API P95 Budget: 250ms"] --> Edge["CDN / TLS / API Gateway<br/>20ms"]
    Edge --> Auth["Auth Token Validation<br/>15ms (Redis / JWT)"]
    Auth --> App["Order Service Logic<br/>35ms"]
    App --> Inv["Inventory RPC<br/>50ms"]
    App --> DB["Primary SQL Commit<br/>100ms"]
    DB --> Buffer["Serialization & Network Slack<br/>30ms"]

When a Gatling test breaches the 250 ms P95 target, distributed traces immediately show which tier overspent its latency budget!


4. Industry Reference Targets by Workload Class

Workload Class P95 SLO P99 SLO P99.9 SLO Success Rate Key Nuance
User-Facing Web API < 300 ms < 800 ms < 2,500 ms $\ge 99.5\%$ Human perception threshold is ~100–300 ms
Mobile BFF API < 200 ms < 500 ms < 1,500 ms $\ge 99.9\%$ Cellular RTT adds 40–100 ms on top of server time
Internal Core RPC < 75 ms < 200 ms < 500 ms $\ge 99.95\%$ Called multiple times per external request
Real-Time / FinTech < 25 ms < 75 ms < 150 ms $\ge 99.99\%$ Requires GC-tuned JVM (ZGC) & warm pools
Kafka Producer Ack < 50 ms < 150 ms < 500 ms $\ge 99.99\%$ Depends on acks=all, linger.ms & batch size
Kafka End-to-End Lag < 500 ms < 2,000 ms < 5,000 ms $\ge 99.9\%$ Consumer throughput must exceed peak produce rate

Common SLO Anti-Patterns

  • Unqualified Latency Targets: Writing "Response time must be < 300 ms" without specifying which percentile (P95? P99? Max?), at what load (500 RPS?), and over what window (15 min?). Demand 100% < 300 ms (Max < 300 ms) is virtually impossible over TCP/IP due to rare packet retransmits!
  • Chasing Five Nines Without Business Need: Moving from 99.9% to 99.999% typically increases infrastructure and engineering cost by 10x+ (active-active multi-region, zero-downtime schema migrations, automated chaos engineering). Match your "nines" to real user impact—users on a residential ISP with 99.9% connectivity cannot even perceive 99.999% server availability!
  • Ignoring P99 When P95 Passes: For high-value customers (who have the largest carts, biggest portfolios, and most order history!), their database queries are the heaviest—meaning your top 1% revenue customers disproportionately live in your P99 latency bucket!

Knowledge Check

#

Your public checkout endpoint synchronously calls 4 internal microservices, and each service has a 99.9% availability rate. Assuming independent failures, what is the maximum composite availability of the checkout endpoint?

#

Why do highest-value enterprise or power-user accounts often experience P99 latency rather than P50 latency?


Next Steps