Skip to content

Key Metrics & Observability Frameworks

Performance metrics translate raw system behavior into actionable engineering signals. Choosing the wrong aggregation—such as tracking average response time or averaging percentiles across nodes—can make a severely degraded system look completely healthy.


1. Anatomy of Request Latency

End-to-end client latency (what Gatling measures) is composed of multiple distinct phases across the network and server stack:

sequenceDiagram
    participant C as Client (Gatling)
    participant N as Network / LB / TLS
    participant A as App Server
    participant D as Database / Cache

    Note over C,N: 1. DNS + TCP SYN/ACK (1 RTT)
    Note over C,N: 2. TLS 1.3 Handshake (1 RTT on new conn)
    C->>A: 3. HTTP Request Sent
    Note over A: Queue Wait (Thread Pool)
    A->>D: 4. Query / Downstream Call
    D-->>A: Result
    Note over A: Business Logic + JSON Serialization
    A-->>C: 5. First Byte Arrives (TTFB)
    A-->>C: 6. Response Body Transfer Complete
Phase What It Measures High Value Indicates
TCP Connect Time Time to complete SYN → SYN-ACK → ACK Network distance, packet loss, LB backlog (somaxconn)
TLS Handshake Cipher negotiation & certificate verification Missing HTTP Keep-Alive / connection reuse, CPU saturation on TLS termination
TTFB (Time to First Byte) Request sent $\rightarrow$ first byte of HTTP response header Server queue wait, slow DB queries, GC pauses, downstream RPC latency
Content Download First byte $\rightarrow$ last byte of response body Huge uncompressed JSON payloads, NIC bandwidth saturation

2. Percentiles, Histograms & Tail Latency

Why Mean (Average) Latency Lies

Consider 10 requests where 9 hit Redis in 50 ms and 1 hits an unindexed SQL query in 5,000 ms: - Mean: 545 ms (describes zero actual users—too high for the fast 90%, 10x too optimistic for the slow 10%!) - Median (P50): 50 ms (completely hides the 5-second stall) - P95 / P99: Immediately exposes the 5,000 ms tail!

Percentile Meaning Typical Engineering Role
P50 (Median) 50% of requests are faster, 50% slower Typical happy-path user experience (cache hit)
P90 / P95 90–95% of requests finish within this bound Primary interactive SLO target
P99 Slowest 1% boundary (1 in 100 requests) Heavy users, cache misses, minor GC pauses; critical for microservice fan-out
P99.9 Slowest 0.1% boundary (1 in 1,000 requests) Stop-The-World Full GC, lock timeouts, packet retransmissions

Mathematical Trap: Never Average Percentiles!

You cannot take the mean or max of P99 values across multiple servers or 1-minute windows!

Suppose Pod A serves 10,000 requests with P99 = 100 ms, and Pod B (stuck in a routing blackhole or health-check trickle) serves 10 requests with P99 = 4,000 ms. - Naive average of P99s: (100 + 4000) / 2 = 2,050 ms (Completely wrong!) - True fleet-wide P99 across all 10,010 requests: ~100 ms!

Conversely, if Pod B received 50% of traffic and stalled, unweighted averaging hides half the damage. Modern tools (Gatling, Prometheus, Datadog DDSketch, HdrHistogram) record logarithmic latency buckets and compute percentiles over the merged histogram.


3. Apdex (Application Performance Index)

When business stakeholders want a single 0.00 to 1.00 user satisfaction score anchored to a target latency threshold $T$, industry teams use Apdex:

  • Satisfied ($S$): Response time $\le T$
  • Tolerating ($T_{\text{tol}}$): Response time between $T$ and $4T$
  • Frustrated ($F$): Response time $> 4T$ OR any HTTP error (5xx / timeout)
Apdex_T = ( Satisfied + 0.5 × Tolerating ) / Total Requests
Apdex Score Range User Experience Rating
0.94 – 1.00 Excellent
0.85 – 0.93 Good
0.70 – 0.84 Fair (optimization needed)
< 0.70 Poor / Unacceptable

4. The Three Canonical Observability Frameworks

During a load test, pair Gatling's client-side metrics with structured server-side telemetry using these three industry-standard frameworks:

Best for end-to-end service health & alerting:

  1. Latency: Time to service requests—crucially separating successful (2xx) latency from failed (5xx) latency! A database crash that returns HTTP 500 in 1 ms will artificially lower overall latency if you mix errors with successes!
  2. Traffic: Demand placed on the system (HTTP RPS, Kafka messages/sec, active sessions).
  3. Errors: Rate of failed requests—explicit (HTTP 5xx, TCP resets), implicit (HTTP 200 with corrupt/empty payload, caught by Gatling check()), and policy breaches (requests slower than SLA).
  4. Saturation: How "full" your most constrained resource is (queue depth, thread pool utilization, heap occupancy, HikariCP active/max ratio).

Created by Tom Wilkie for request-driven microservices (measure on every endpoint):

  • R — Rate: Requests per second (RPS) handled by the service.
  • E — Errors: Number/percentage of failed requests per second.
  • D — Duration: Histogram distribution (P50, P95, P99) of request durations.

Notice: Gatling's HTML report natively reports RED for every request transaction!

Created by Brendan Gregg for hardware & OS resources (check every CPU, disk, NIC, mutex, pool):

  • U — Utilization: % of time the resource was busy servicing work (e.g., 78% CPU, 90% disk busy).
  • S — Saturation: Degree of queued extra work that cannot be serviced immediately (e.g., Linux CPU run-queue length > core count, HikariCP threadsAwaitingConnection > 0, disk aqu-sz).
  • E — Errors: Count of hardware/driver/allocation error events (NIC dropped packets, TCP retransmits, OOM kills).
flowchart LR
    Gatling["Gatling Client Report<br/>RED: Rate • Errors • Duration"] <-->|"Correlate by Timestamp & Tags"| APM["Server APM & Host Telemetry<br/>Golden Signals + USE Method"]
    APM --> CPU["CPU: Util% & Run Queue"]
    APM --> Mem["JVM: Heap, Allocation & GC Pause"]
    APM --> DB["DB Pool: Active vs Waiting"]
    APM --> Net["Network: Bandwidth & Retransmits"]

5. Throughput vs. Goodput

  • Raw Throughput (RPS): Total HTTP responses completed per second (including fast 429 or 500 errors!).
  • Goodput: Successful, SLA-compliant requests per second (2xx responses passing all functional checks within latency targets).
  • Transactions Per Second (TPS): Complete multi-step business workflows per second (e.g., Search → View → Cart → Checkout = 4 HTTP requests, so 1,000 RPS = 250 TPS).

Beware the 'Fast Error' Throughput Mirage

If your database crashes mid-test and your API immediately returns HTTP 500 in 0.5 ms, raw RPS can jump from 1,000 RPS to 25,000 RPS while mean latency drops to 1 ms! Always assert on successful request percentage and evaluate OK response times separately from KO response times in Gatling reports.


Target Benchmarks by System Archetype

System Archetype P95 Target P99 Target Error Budget (Success %) Key Saturation Metric
Public Web / E-Commerce < 300 ms < 800 ms > 99.5% DB pool wait & cache hit %
Mobile BFF / Gateway API < 200 ms < 500 ms > 99.9% Downstream fan-out tail latency
Internal East-West RPC < 50–100 ms < 250 ms > 99.95% Thread/event-loop saturation
Low-Latency Trading / AdTech < 10–25 ms < 50 ms > 99.99% JVM GC pauses, NIC interrupts
Kafka Event Ingestion < 50 ms (ack) < 200 ms > 99.99% Broker request queue & consumer lag

Knowledge Check

#

Why do Google SRE's Four Golden Signals insist on measuring successful (2xx) latency separately from failed (5xx) latency?

#

In Brendan Gregg's USE Method, which metric directly measures Saturation of a database connection pool?


Next Steps