Key Metrics & Observability Frameworks¶
Performance metrics translate raw system behavior into actionable engineering signals. Choosing the wrong aggregation—such as tracking average response time or averaging percentiles across nodes—can make a severely degraded system look completely healthy.
1. Anatomy of Request Latency¶
End-to-end client latency (what Gatling measures) is composed of multiple distinct phases across the network and server stack:
sequenceDiagram
participant C as Client (Gatling)
participant N as Network / LB / TLS
participant A as App Server
participant D as Database / Cache
Note over C,N: 1. DNS + TCP SYN/ACK (1 RTT)
Note over C,N: 2. TLS 1.3 Handshake (1 RTT on new conn)
C->>A: 3. HTTP Request Sent
Note over A: Queue Wait (Thread Pool)
A->>D: 4. Query / Downstream Call
D-->>A: Result
Note over A: Business Logic + JSON Serialization
A-->>C: 5. First Byte Arrives (TTFB)
A-->>C: 6. Response Body Transfer Complete
| Phase | What It Measures | High Value Indicates |
|---|---|---|
| TCP Connect Time | Time to complete SYN → SYN-ACK → ACK |
Network distance, packet loss, LB backlog (somaxconn) |
| TLS Handshake | Cipher negotiation & certificate verification | Missing HTTP Keep-Alive / connection reuse, CPU saturation on TLS termination |
| TTFB (Time to First Byte) | Request sent $\rightarrow$ first byte of HTTP response header | Server queue wait, slow DB queries, GC pauses, downstream RPC latency |
| Content Download | First byte $\rightarrow$ last byte of response body | Huge uncompressed JSON payloads, NIC bandwidth saturation |
2. Percentiles, Histograms & Tail Latency¶
Why Mean (Average) Latency Lies¶
Consider 10 requests where 9 hit Redis in 50 ms and 1 hits an unindexed SQL query in 5,000 ms:
- Mean: 545 ms (describes zero actual users—too high for the fast 90%, 10x too optimistic for the slow 10%!)
- Median (P50): 50 ms (completely hides the 5-second stall)
- P95 / P99: Immediately exposes the 5,000 ms tail!
| Percentile | Meaning | Typical Engineering Role |
|---|---|---|
| P50 (Median) | 50% of requests are faster, 50% slower | Typical happy-path user experience (cache hit) |
| P90 / P95 | 90–95% of requests finish within this bound | Primary interactive SLO target |
| P99 | Slowest 1% boundary (1 in 100 requests) | Heavy users, cache misses, minor GC pauses; critical for microservice fan-out |
| P99.9 | Slowest 0.1% boundary (1 in 1,000 requests) | Stop-The-World Full GC, lock timeouts, packet retransmissions |
Mathematical Trap: Never Average Percentiles!
You cannot take the mean or max of P99 values across multiple servers or 1-minute windows!
Suppose Pod A serves 10,000 requests with P99 = 100 ms, and Pod B (stuck in a routing blackhole or health-check trickle) serves 10 requests with P99 = 4,000 ms.
- Naive average of P99s: (100 + 4000) / 2 = 2,050 ms (Completely wrong!)
- True fleet-wide P99 across all 10,010 requests: ~100 ms!
Conversely, if Pod B received 50% of traffic and stalled, unweighted averaging hides half the damage. Modern tools (Gatling, Prometheus, Datadog DDSketch, HdrHistogram) record logarithmic latency buckets and compute percentiles over the merged histogram.
3. Apdex (Application Performance Index)¶
When business stakeholders want a single 0.00 to 1.00 user satisfaction score anchored to a target latency threshold $T$, industry teams use Apdex:
- Satisfied ($S$): Response time $\le T$
- Tolerating ($T_{\text{tol}}$): Response time between $T$ and $4T$
- Frustrated ($F$): Response time $> 4T$ OR any HTTP error (5xx / timeout)
| Apdex Score Range | User Experience Rating |
|---|---|
| 0.94 – 1.00 | Excellent |
| 0.85 – 0.93 | Good |
| 0.70 – 0.84 | Fair (optimization needed) |
| < 0.70 | Poor / Unacceptable |
4. The Three Canonical Observability Frameworks¶
During a load test, pair Gatling's client-side metrics with structured server-side telemetry using these three industry-standard frameworks:
Best for end-to-end service health & alerting:
- Latency: Time to service requests—crucially separating successful (2xx) latency from failed (5xx) latency! A database crash that returns
HTTP 500in1 mswill artificially lower overall latency if you mix errors with successes! - Traffic: Demand placed on the system (HTTP RPS, Kafka messages/sec, active sessions).
- Errors: Rate of failed requests—explicit (
HTTP 5xx, TCP resets), implicit (HTTP 200with corrupt/empty payload, caught by Gatlingcheck()), and policy breaches (requests slower than SLA). - Saturation: How "full" your most constrained resource is (queue depth, thread pool utilization, heap occupancy, HikariCP active/max ratio).
Created by Tom Wilkie for request-driven microservices (measure on every endpoint):
- R — Rate: Requests per second (
RPS) handled by the service. - E — Errors: Number/percentage of failed requests per second.
- D — Duration: Histogram distribution (
P50,P95,P99) of request durations.
Notice: Gatling's HTML report natively reports RED for every request transaction!
Created by Brendan Gregg for hardware & OS resources (check every CPU, disk, NIC, mutex, pool):
- U — Utilization: % of time the resource was busy servicing work (e.g., 78% CPU, 90% disk busy).
- S — Saturation: Degree of queued extra work that cannot be serviced immediately (e.g., Linux CPU run-queue length > core count, HikariCP
threadsAwaitingConnection > 0, diskaqu-sz). - E — Errors: Count of hardware/driver/allocation error events (NIC dropped packets, TCP retransmits, OOM kills).
flowchart LR
Gatling["Gatling Client Report<br/>RED: Rate • Errors • Duration"] <-->|"Correlate by Timestamp & Tags"| APM["Server APM & Host Telemetry<br/>Golden Signals + USE Method"]
APM --> CPU["CPU: Util% & Run Queue"]
APM --> Mem["JVM: Heap, Allocation & GC Pause"]
APM --> DB["DB Pool: Active vs Waiting"]
APM --> Net["Network: Bandwidth & Retransmits"]
5. Throughput vs. Goodput¶
- Raw Throughput (RPS): Total HTTP responses completed per second (including fast
429or500errors!). - Goodput: Successful, SLA-compliant requests per second (
2xxresponses passing all functional checks within latency targets). - Transactions Per Second (TPS): Complete multi-step business workflows per second (e.g.,
Search → View → Cart → Checkout = 4 HTTP requests, so1,000 RPS = 250 TPS).
Beware the 'Fast Error' Throughput Mirage
If your database crashes mid-test and your API immediately returns HTTP 500 in 0.5 ms, raw RPS can jump from 1,000 RPS to 25,000 RPS while mean latency drops to 1 ms! Always assert on successful request percentage and evaluate OK response times separately from KO response times in Gatling reports.
Target Benchmarks by System Archetype¶
| System Archetype | P95 Target | P99 Target | Error Budget (Success %) | Key Saturation Metric |
|---|---|---|---|---|
| Public Web / E-Commerce | < 300 ms | < 800 ms | > 99.5% | DB pool wait & cache hit % |
| Mobile BFF / Gateway API | < 200 ms | < 500 ms | > 99.9% | Downstream fan-out tail latency |
| Internal East-West RPC | < 50–100 ms | < 250 ms | > 99.95% | Thread/event-loop saturation |
| Low-Latency Trading / AdTech | < 10–25 ms | < 50 ms | > 99.99% | JVM GC pauses, NIC interrupts |
| Kafka Event Ingestion | < 50 ms (ack) | < 200 ms | > 99.99% | Broker request queue & consumer lag |
Knowledge Check¶
Why do Google SRE's Four Golden Signals insist on measuring successful (2xx) latency separately from failed (5xx) latency?
In Brendan Gregg's USE Method, which metric directly measures Saturation of a database connection pool?
Next Steps¶
- Read next: Mathematical Laws & Queuing Theory — Learn Little's Law, M/M/1 queuing curves, Amdahl's Law, and the Universal Scalability Law.