Skip to content

Bottlenecks, Resilience & Capacity Planning

Load testing tells you that latency degraded at 1,500 RPS; bottleneck diagnostics tells you why, resilience patterns keep overload from becoming a total outage, and capacity planning translates test numbers into production infrastructure decisions.


1. Systematic Bottleneck Taxonomy

Every request traverses four hardware domains (CPU, Memory, Disk/Storage, Network) plus Software Contention (locks, thread pools, connection pools).

flowchart TD
    Symptom["Symptom: P95 / P99 Latency Spike or Throughput Plateau"] --> Q1{"Is CPU > 85%?"}
    Q1 -->|Yes| CPU["CPU-Bound<br/>• Hot loops / regex / JSON serialization<br/>• TLS handshakes / compression<br/>• High GC CPU or kernel context switching"]
    Q1 -->|No| Q2{"Are Worker / DB Pools 100% Busy?"}
    Q2 -->|Yes| Contention["Software Contention / Downstream Wait<br/>• DB connection pool exhausted<br/>• Slow SQL / row lock wait<br/>• Slow external HTTP / RPC dependency"]
    Q2 -->|No| Q3{"Periodic Latency Spikes Every N Seconds?"}
    Q3 -->|Yes| GC["Memory / GC / Flush Stalls<br/>• Stop-The-World JVM GC pauses<br/>• Disk page-cache / WAL checkpoint flush<br/>• Cache TTL expiration stampede"]
    Q3 -->|No| Net["Network / OS Limits<br/>• Ephemeral port exhaustion (TIME_WAIT)<br/>• NIC bandwidth ceiling<br/>• Load-generator saturation"]

Layer-by-Layer Diagnostic Guide

  • High User CPU (%usr): Serialization (Jackson/Protobuf), cryptographic hashing (BCrypt/TLS), regex backtracking, inefficient $O(N^2)$ collection scans. Profile with Java Flight Recorder (JFR) or async-profiler flame graphs.
  • High System/Kernel CPU (%sys): Excessive OS thread context switching (thousands of blocking platform threads), heavy syscall/socket churn without connection pooling.
  • Low CPU + High Latency: Classic signature of lock contention (synchronized, ReentrantLock) or blocking I/O wait (threads parked waiting on DB or downstream REST APIs).
  • Allocation Rate Pressure: Allocating gigabytes/sec of short-lived DTOs triggers frequent Young GC pauses.
  • Old-Gen Promotion / Humongous Objects: Large JSON payloads (>50% of a G1 region) allocate directly as Humongous objects, causing premature evacuation stalls.
  • Memory Leak Signature: In a Soak Test, post-GC baseline heap usage climbs in a sawtooth ramp that never returns to its prior floor, culminating in OutOfMemoryError or back-to-back Full GC thrashing.
  • Modern Collectors: For low-tail-latency Java 21+ services, compare G1GC (-XX:+UseG1GC -XX:MaxGCPauseMillis=100) against Generational ZGC (-XX:+UseZGC -XX:+ZGenerational), which keeps Stop-The-World pauses sub-millisecond.
  • Oversized Connection Pools: More DB connections $\neq$ higher throughput! PostgreSQL and MySQL suffer severe context-switch and latch contention when hundreds of active queries compete for a handful of CPU cores.
  • HikariCP Pool Sizing Formula:
    Optimal Pool Size ≈ (CPU Core Count × 2) + Effective Spindle Count
    For a 4-core SSD database instance, 10–15 active connections per DB node often outperforms 200 connections!
  • Lock Contention: Hot-row UPDATE statements (e.g., decrementing inventory for a single flash-sale SKU) serialize transactions via Amdahl's Law even when DB CPU is 15%.
  • Missing HTTP Keep-Alive: Opening a fresh TCP + TLS 1.3 connection per request adds 2–3 RTTs of latency and rapidly exhausts the OS ephemeral port range (net.ipv4.ip_local_port_range, often ~28,000 ports) as closed sockets sit in TIME_WAIT for 60 seconds!
  • HTTP/1.1 vs. HTTP/2: HTTP/1.1 requires multiple connections to avoid head-of-line blocking; HTTP/2 multiplexes concurrent streams over a single persistent connection (watch out for SETTINGS_MAX_CONCURRENT_STREAMS ceilings).

2. Caching Failure Modes Under Load

Caches routinely mask database limits during naive load tests—and cause catastrophic production outages when they fail under real traffic patterns.

Failure Mode Mechanism How to Catch & Mitigate
Cache Stampede (Thundering Herd) A hot cached key expires while 2,000 concurrent requests are reading it; all 2,000 miss cache simultaneously and hammer the DB with identical expensive queries Single-flight / request coalescing (mutex lock on recompute), probabilistic early expiration (XFetch), stale-while-revalidate
Cache Avalanche Thousands of keys populated at startup with an identical TTL = 3600s expire at the exact same second Add random TTL jitter (TTL = 3600 + random(0, 300))
Cache Penetration Requests repeatedly query non-existent IDs (productId=9999999), bypassing cache on every call and hitting DB Cache negative lookups briefly (null sentinel with short TTL) or guard with a Bloom filter
99.9% Cache-Hit Test Illusion A Gatling test requests GET /products/1 in a tight loop without a feeder, testing Redis RAM instead of real catalog distribution Feed realistic Zipfian / CSV distributions and test both cold-cache and warm-cache states

3. Overload Protection & Resilience Patterns

When incoming open-model traffic exceeds maximum capacity ($\lambda > \mu$), an unprotected server accepts every request into unbounded queues until latency hits 30+ seconds, clients time out, and retries multiply the load by 3–5x (Retry Storm)!

flowchart LR
    Traffic["Incoming Surge (5,000 RPS)"] --> RL["Rate Limiter / Load Shedder<br/>Reject excess fast with HTTP 429 / 503"]
    RL -->|"2,000 RPS admitted"| BH["Bulkhead Isolation<br/>Separate pools per tenant / endpoint"]
    BH --> Service["Core Service<br/>P95 stays < 250ms for admitted users!"]
    Service --> CB["Circuit Breaker + Bounded Timeout<br/>Fail fast if downstream stalls"]
    CB --> DB[("Database")]

Essential Stability Mechanisms to Validate in Spike & Stress Tests

  1. Bounded Timeouts Everywhere: Never use infinite default socket/connection timeouts. Set client timeouts slightly above downstream P99.9 so stalled threads are reclaimed promptly.
  2. Load Shedding & Rate Limiting: Returning HTTP 429 Too Many Requests or 503 Service Unavailable in 2 ms for excess 20% traffic preserves 200 ms latency for the 80% of traffic you can serve—infinitely better than timing out 100% of users!
  3. Circuit Breakers: Trip open when downstream error/slow-call rate exceeds threshold (e.g., >50% over 10s), returning cached fallbacks or degraded responses instead of exhausting caller threads.
  4. Bulkheads: Isolate thread pools or semaphores by criticality (e.g., prevent a slow /reports/export endpoint from starving /checkout).
  5. Exponential Backoff + Jitter: Retries without jitter synchronize into destructive waves. Always validate that client retry policies include full jitter and a retry budget (e.g., max 10% extra retry traffic).

4. Capacity Planning Methodology

Capacity planning answers: "How many instances/shards do we need to survive peak traffic with headroom for node failures?"

Step 1: Find the Knee & Maximum Sustainable Throughput (MST)

Run a Step Load Test on a single production-sized instance (or known reference cluster) and measure: - Knee of the Curve: Load where latency shifts from flat/linear to super-linear queuing. - Maximum Sustainable Throughput (MST): Highest RPS where all SLOs (P95, P99, Error Rate < 0.1%, CPU < 75%) hold steady for at least 15–30 minutes.

Step 2: Calculate Production Fleet Size

Required Instances = ceil( (Peak Forecast RPS × Safety Factor) / (Single-Node MST × Usable Headroom) ) + Failover Nodes (N + k)

Worked Example: - Forecast Black Friday peak: 6,000 RPS - Safety margin for forecast uncertainty / micro-bursts: 1.25x $\rightarrow$ 7,500 RPS target - Single pod MST (while meeting P99 < 300ms at 70% CPU): 750 RPS - Base active pods needed: $7500 / 750 = \mathbf{10\text{ pods}}$ - Multi-AZ failover ($N + 2$ so losing 1 AZ or 2 pods keeps fleet safe): 12 pods minimum

Step 3: Account for Auto-Scaling Lag!

Auto-Scaling Does Not Save You From Instant Spikes

Kubernetes HPA / Cloud Auto-Scalers have a reaction pipeline delay: 1. Metric scrape & window evaluation: 15–60s 2. Pod scheduling & node provisioning: 10–90s 3. Container pull, JVM startup, Spring context init & JIT warmup: 20–60s

Total lag = 45 seconds to 3+ minutes! During a Spike Test, your pre-scaled baseline fleet (plus queues/load shedding) must survive those first 2 minutes without crashing.


Knowledge Check

#

During a load test, Gatling reports P95 latency jumping from 80 ms to 2,400 ms, while API server CPU stays low at 18%. What is the most likely root cause?

#

What happens during a "Cache Stampede" (Thundering Herd)?

#

Why can Kubernetes HPA / cloud auto-scaling still fail a Spike Test even when max replicas is set high?


Next Steps