SLIs, SLOs, SLAs & Error Budgets¶
Modern reliability and performance engineering distinguishes sharply between what you measure (SLI), what your engineering team targets internally (SLO), and what your business legally promises customers (SLA). Conflating the three leads either to alert fatigue or costly contract breaches.
1. The Reliability Hierarchy: SLI → SLO → SLA¶
flowchart LR
SLI["SLI (Indicator)<br/>Quantitative measurement<br/>e.g., P95 of valid 2xx responses = 182ms"] --> SLO["SLO (Objective)<br/>Internal engineering target<br/>e.g., P95 < 250ms over 28 days"]
SLO --> EB["Error Budget<br/>100% - SLO<br/>Room for releases, spikes & experiments"]
SLO --> SLA["SLA (Agreement)<br/>External contract + financial credits<br/>e.g., P95 < 500ms, Availability >= 99.9%"]
| Layer | Full Name | Owner | Example | Consequence if Breached |
|---|---|---|---|---|
| SLI | Service Level Indicator | Engineering / SRE | % of HTTP requests completing < 250ms with non-5xx status |
None by itself—it is the speedometer |
| SLO | Service Level Objective | Product + Engineering | 99.5% of requests in 28-day window satisfy the latency SLI |
Error budget depleted; freeze risky feature rollouts, prioritize performance/reliability work |
| SLA | Service Level Agreement | Legal / Sales + Leadership | Monthly uptime >= 99.9% & P95 < 500ms |
Financial refunds / service credits, reputational damage, contract termination |
Golden Rule: Set Internal SLOs Stricter Than External SLAs
If your customer contract (SLA) promises P95 < 500 ms and 99.9% availability, set your internal SLO (and Gatling CI assertions!) at P95 < 300 ms and 99.95%. That margin gives your team early warning to fix regressions before customers or contracts are impacted.
2. Availability "Nines" & Error Budgets¶
An Error Budget is 100% - SLO. It quantifies the exact volume of failed/slow requests or downtime permissible within a rolling window:
| Target ("Nines") | Error Budget % | Allowable Downtime / Month | Allowable Downtime / Year | Failed Requests per 10M Requests |
|---|---|---|---|---|
| 99% ("Two nines") | 1.000% | 7 hr 18 min | 3.65 days | 100,000 |
| 99.5% | 0.500% | 3 hr 39 min | 1.83 days | 50,000 |
| 99.9% ("Three nines") | 0.100% | 43 min 50 sec | 8 hr 46 min | 10,000 |
| 99.95% | 0.050% | 21 min 55 sec | 4 hr 23 min | 5,000 |
| 99.99% ("Four nines") | 0.010% | 4 min 23 sec | 52 min 36 sec | 1,000 |
| 99.999% ("Five nines") | 0.001% | 26.3 seconds | 5 min 15 sec | 100 |
Request-Based Error Budgets in Load Testing¶
Time-based uptime hides partial brownouts! Always express load-test SLOs as request-based budgets:
In Gatling Java DSL, encode both availability and percentile SLOs directly as build-breaking .assertions():
setUp(
checkoutScenario.injectOpen(constantUsersPerSec(200).during(Duration.ofMinutes(15)))
)
.protocols(httpProtocol)
.assertions(
global().responseTime().percentile(95.0).lt(300), // Latency SLO (stricter than 500ms SLA)
global().responseTime().percentile(99.0).lt(800), // Tail Latency SLO
global().successfulRequests().percent().gte(99.9), // Error Budget <= 0.1%
forAll().responseTime().percentile(99.0).lt(1200) // Every individual endpoint < 1.2s
);
3. Composite Availability & Latency Budgeting in Microservices¶
Serial Dependency Availability Math¶
If a single POST /checkout call synchronously invokes 5 microservices in series (Gateway $\rightarrow$ Auth $\rightarrow$ Cart $\rightarrow$ Pricing $\rightarrow$ Payment), and each service individually achieves 99.9% availability:
To guarantee a 99.9% public API SLO across 5 synchronous hops, each internal downstream dependency must achieve $\ge 99.98\%$—or be decoupled asynchronously (e.g., via Kafka or cached fallbacks).
End-to-End Latency Budgeting¶
Allocate your public P95 latency budget explicitly across architectural tiers before load testing:
flowchart LR
Total["Public API P95 Budget: 250ms"] --> Edge["CDN / TLS / API Gateway<br/>20ms"]
Edge --> Auth["Auth Token Validation<br/>15ms (Redis / JWT)"]
Auth --> App["Order Service Logic<br/>35ms"]
App --> Inv["Inventory RPC<br/>50ms"]
App --> DB["Primary SQL Commit<br/>100ms"]
DB --> Buffer["Serialization & Network Slack<br/>30ms"]
When a Gatling test breaches the 250 ms P95 target, distributed traces immediately show which tier overspent its latency budget!
4. Industry Reference Targets by Workload Class¶
| Workload Class | P95 SLO | P99 SLO | P99.9 SLO | Success Rate | Key Nuance |
|---|---|---|---|---|---|
| User-Facing Web API | < 300 ms | < 800 ms | < 2,500 ms | $\ge 99.5\%$ | Human perception threshold is ~100–300 ms |
| Mobile BFF API | < 200 ms | < 500 ms | < 1,500 ms | $\ge 99.9\%$ | Cellular RTT adds 40–100 ms on top of server time |
| Internal Core RPC | < 75 ms | < 200 ms | < 500 ms | $\ge 99.95\%$ | Called multiple times per external request |
| Real-Time / FinTech | < 25 ms | < 75 ms | < 150 ms | $\ge 99.99\%$ | Requires GC-tuned JVM (ZGC) & warm pools |
| Kafka Producer Ack | < 50 ms | < 150 ms | < 500 ms | $\ge 99.99\%$ | Depends on acks=all, linger.ms & batch size |
| Kafka End-to-End Lag | < 500 ms | < 2,000 ms | < 5,000 ms | $\ge 99.9\%$ | Consumer throughput must exceed peak produce rate |
Common SLO Anti-Patterns¶
- Unqualified Latency Targets: Writing "Response time must be < 300 ms" without specifying which percentile (
P95?P99?Max?), at what load (500 RPS?), and over what window (15 min?). Demand100% < 300 ms(Max < 300 ms) is virtually impossible over TCP/IP due to rare packet retransmits! - Chasing Five Nines Without Business Need: Moving from
99.9%to99.999%typically increases infrastructure and engineering cost by 10x+ (active-active multi-region, zero-downtime schema migrations, automated chaos engineering). Match your "nines" to real user impact—users on a residential ISP with99.9%connectivity cannot even perceive99.999%server availability! - Ignoring P99 When P95 Passes: For high-value customers (who have the largest carts, biggest portfolios, and most order history!), their database queries are the heaviest—meaning your top 1% revenue customers disproportionately live in your P99 latency bucket!
Knowledge Check¶
Your public checkout endpoint synchronously calls 4 internal microservices, and each service has a 99.9% availability rate. Assuming independent failures, what is the maximum composite availability of the checkout endpoint?
Why do highest-value enterprise or power-user accounts often experience P99 latency rather than P50 latency?
Next Steps¶
- Read next: Bottlenecks, Resilience & Capacity Planning — Diagnose CPU, GC, DB, cache, and network bottlenecks and calculate fleet capacity.