Bottlenecks, Resilience & Capacity Planning¶
Load testing tells you that latency degraded at 1,500 RPS; bottleneck diagnostics tells you why, resilience patterns keep overload from becoming a total outage, and capacity planning translates test numbers into production infrastructure decisions.
1. Systematic Bottleneck Taxonomy¶
Every request traverses four hardware domains (CPU, Memory, Disk/Storage, Network) plus Software Contention (locks, thread pools, connection pools).
flowchart TD
Symptom["Symptom: P95 / P99 Latency Spike or Throughput Plateau"] --> Q1{"Is CPU > 85%?"}
Q1 -->|Yes| CPU["CPU-Bound<br/>• Hot loops / regex / JSON serialization<br/>• TLS handshakes / compression<br/>• High GC CPU or kernel context switching"]
Q1 -->|No| Q2{"Are Worker / DB Pools 100% Busy?"}
Q2 -->|Yes| Contention["Software Contention / Downstream Wait<br/>• DB connection pool exhausted<br/>• Slow SQL / row lock wait<br/>• Slow external HTTP / RPC dependency"]
Q2 -->|No| Q3{"Periodic Latency Spikes Every N Seconds?"}
Q3 -->|Yes| GC["Memory / GC / Flush Stalls<br/>• Stop-The-World JVM GC pauses<br/>• Disk page-cache / WAL checkpoint flush<br/>• Cache TTL expiration stampede"]
Q3 -->|No| Net["Network / OS Limits<br/>• Ephemeral port exhaustion (TIME_WAIT)<br/>• NIC bandwidth ceiling<br/>• Load-generator saturation"]
Layer-by-Layer Diagnostic Guide¶
- High User CPU (
%usr): Serialization (Jackson/Protobuf), cryptographic hashing (BCrypt/TLS), regex backtracking, inefficient $O(N^2)$ collection scans. Profile with Java Flight Recorder (JFR) or async-profiler flame graphs. - High System/Kernel CPU (
%sys): Excessive OS thread context switching (thousands of blocking platform threads), heavy syscall/socket churn without connection pooling. - Low CPU + High Latency: Classic signature of lock contention (
synchronized,ReentrantLock) or blocking I/O wait (threads parked waiting on DB or downstream REST APIs).
- Allocation Rate Pressure: Allocating gigabytes/sec of short-lived DTOs triggers frequent Young GC pauses.
- Old-Gen Promotion / Humongous Objects: Large JSON payloads (>50% of a G1 region) allocate directly as Humongous objects, causing premature evacuation stalls.
- Memory Leak Signature: In a Soak Test, post-GC baseline heap usage climbs in a sawtooth ramp that never returns to its prior floor, culminating in
OutOfMemoryErroror back-to-back Full GC thrashing. - Modern Collectors: For low-tail-latency Java 21+ services, compare G1GC (
-XX:+UseG1GC -XX:MaxGCPauseMillis=100) against Generational ZGC (-XX:+UseZGC -XX:+ZGenerational), which keeps Stop-The-World pauses sub-millisecond.
- Oversized Connection Pools: More DB connections $\neq$ higher throughput! PostgreSQL and MySQL suffer severe context-switch and latch contention when hundreds of active queries compete for a handful of CPU cores.
- HikariCP Pool Sizing Formula:
Optimal Pool Size ≈ (CPU Core Count × 2) + Effective Spindle CountFor a 4-core SSD database instance, 10–15 active connections per DB node often outperforms 200 connections!
- Lock Contention: Hot-row
UPDATEstatements (e.g., decrementing inventory for a single flash-sale SKU) serialize transactions via Amdahl's Law even when DB CPU is 15%.
- Missing HTTP Keep-Alive: Opening a fresh TCP + TLS 1.3 connection per request adds 2–3 RTTs of latency and rapidly exhausts the OS ephemeral port range (
net.ipv4.ip_local_port_range, often ~28,000 ports) as closed sockets sit inTIME_WAITfor 60 seconds! - HTTP/1.1 vs. HTTP/2: HTTP/1.1 requires multiple connections to avoid head-of-line blocking; HTTP/2 multiplexes concurrent streams over a single persistent connection (watch out for
SETTINGS_MAX_CONCURRENT_STREAMSceilings).
2. Caching Failure Modes Under Load¶
Caches routinely mask database limits during naive load tests—and cause catastrophic production outages when they fail under real traffic patterns.
| Failure Mode | Mechanism | How to Catch & Mitigate |
|---|---|---|
| Cache Stampede (Thundering Herd) | A hot cached key expires while 2,000 concurrent requests are reading it; all 2,000 miss cache simultaneously and hammer the DB with identical expensive queries | Single-flight / request coalescing (mutex lock on recompute), probabilistic early expiration (XFetch), stale-while-revalidate |
| Cache Avalanche | Thousands of keys populated at startup with an identical TTL = 3600s expire at the exact same second |
Add random TTL jitter (TTL = 3600 + random(0, 300)) |
| Cache Penetration | Requests repeatedly query non-existent IDs (productId=9999999), bypassing cache on every call and hitting DB |
Cache negative lookups briefly (null sentinel with short TTL) or guard with a Bloom filter |
| 99.9% Cache-Hit Test Illusion | A Gatling test requests GET /products/1 in a tight loop without a feeder, testing Redis RAM instead of real catalog distribution |
Feed realistic Zipfian / CSV distributions and test both cold-cache and warm-cache states |
3. Overload Protection & Resilience Patterns¶
When incoming open-model traffic exceeds maximum capacity ($\lambda > \mu$), an unprotected server accepts every request into unbounded queues until latency hits 30+ seconds, clients time out, and retries multiply the load by 3–5x (Retry Storm)!
flowchart LR
Traffic["Incoming Surge (5,000 RPS)"] --> RL["Rate Limiter / Load Shedder<br/>Reject excess fast with HTTP 429 / 503"]
RL -->|"2,000 RPS admitted"| BH["Bulkhead Isolation<br/>Separate pools per tenant / endpoint"]
BH --> Service["Core Service<br/>P95 stays < 250ms for admitted users!"]
Service --> CB["Circuit Breaker + Bounded Timeout<br/>Fail fast if downstream stalls"]
CB --> DB[("Database")]
Essential Stability Mechanisms to Validate in Spike & Stress Tests¶
- Bounded Timeouts Everywhere: Never use infinite default socket/connection timeouts. Set client timeouts slightly above downstream P99.9 so stalled threads are reclaimed promptly.
- Load Shedding & Rate Limiting: Returning
HTTP 429 Too Many Requestsor503 Service Unavailablein2 msfor excess 20% traffic preserves200 mslatency for the 80% of traffic you can serve—infinitely better than timing out 100% of users! - Circuit Breakers: Trip open when downstream error/slow-call rate exceeds threshold (e.g., >50% over 10s), returning cached fallbacks or degraded responses instead of exhausting caller threads.
- Bulkheads: Isolate thread pools or semaphores by criticality (e.g., prevent a slow
/reports/exportendpoint from starving/checkout). - Exponential Backoff + Jitter: Retries without jitter synchronize into destructive waves. Always validate that client retry policies include full jitter and a retry budget (e.g., max 10% extra retry traffic).
4. Capacity Planning Methodology¶
Capacity planning answers: "How many instances/shards do we need to survive peak traffic with headroom for node failures?"
Step 1: Find the Knee & Maximum Sustainable Throughput (MST)¶
Run a Step Load Test on a single production-sized instance (or known reference cluster) and measure: - Knee of the Curve: Load where latency shifts from flat/linear to super-linear queuing. - Maximum Sustainable Throughput (MST): Highest RPS where all SLOs (P95, P99, Error Rate < 0.1%, CPU < 75%) hold steady for at least 15–30 minutes.
Step 2: Calculate Production Fleet Size¶
Worked Example: - Forecast Black Friday peak: 6,000 RPS - Safety margin for forecast uncertainty / micro-bursts: 1.25x $\rightarrow$ 7,500 RPS target - Single pod MST (while meeting P99 < 300ms at 70% CPU): 750 RPS - Base active pods needed: $7500 / 750 = \mathbf{10\text{ pods}}$ - Multi-AZ failover ($N + 2$ so losing 1 AZ or 2 pods keeps fleet safe): 12 pods minimum
Step 3: Account for Auto-Scaling Lag!¶
Auto-Scaling Does Not Save You From Instant Spikes
Kubernetes HPA / Cloud Auto-Scalers have a reaction pipeline delay:
1. Metric scrape & window evaluation: 15–60s
2. Pod scheduling & node provisioning: 10–90s
3. Container pull, JVM startup, Spring context init & JIT warmup: 20–60s
Total lag = 45 seconds to 3+ minutes! During a Spike Test, your pre-scaled baseline fleet (plus queues/load shedding) must survive those first 2 minutes without crashing.
Knowledge Check¶
During a load test, Gatling reports P95 latency jumping from 80 ms to 2,400 ms, while API server CPU stays low at 18%. What is the most likely root cause?
What happens during a "Cache Stampede" (Thundering Herd)?
Why can Kubernetes HPA / cloud auto-scaling still fail a Spike Test even when max replicas is set high?
Next Steps¶
- Continue: Common Pitfalls & Shift-Left CI — Avoid classic testing anti-patterns and automate regression gates in CI/CD.