Load Testing Methodology¶
Repeatable, trustworthy performance results come from a scientific, eight-phase engineering methodology—never from firing ad-hoc traffic at an unprepared environment.
flowchart LR
P1["1. Plan<br/>SLOs & Critical Paths"] --> P2["2. Model<br/>Traffic Mix & Data"]
P2 --> P3["3. Implement<br/>Gatling Simulation"]
P3 --> P4["4. Smoke<br/>1-User Validation"]
P4 --> P5["5. Execute<br/>Warm-Up + Target Profile"]
P5 --> P6["6. Analyze<br/>Gatling + APM Traces"]
P6 --> P7["7. Optimize<br/>Single-Variable Fix"]
P7 --> P8["8. Re-Verify<br/>Regression Gate"]
P8 -.->|"Next bottleneck"| P5
Phase 1: Plan & Quantify Objectives¶
Define Measurable Acceptance Criteria¶
Before writing simulation code, align Product, SRE, and Engineering on quantitative pass/fail gates:
| Service Path | Peak Target Load | P95 SLO | P99 SLO | Max Error Rate |
|---|---|---|---|---|
GET /products (Catalog Browse) |
3,500 RPS | < 150 ms | < 350 ms | < 0.05% |
POST /cart (Cart Mutation) |
800 RPS | < 250 ms | < 500 ms | < 0.10% |
POST /checkout (Order Commit) |
200 RPS | < 500 ms | < 1,000 ms | < 0.05% |
Estimate Target Load from Production Telemetry¶
- Historical Peak: Query last peak event RPS (e.g., Black Friday, Monday 9 AM login surge).
- Peak-to-Average Ratio: Web traffic peaks are typically 3x–8x daily average RPS! Never divide daily requests by
86,400without applying a peak multiplier. - Safety Headroom: Validate up to 1.5x–2x forecasted peak so unexpected growth or a single AZ failure stays below the queuing knee.
Phase 2: Design Realistic Workloads & Data¶
Avoid Synthetic Monoculture¶
| Anti-Pattern | Production-Realistic Design |
|---|---|
100% of users hit POST /checkout with zero pauses |
80/20 Funnel: 70% browse/search, 20% add to cart, 10% checkout with randomized pause(3, 12) |
Every iteration queries userId = 1 |
Parameterized Feeders: 50,000+ distinct IDs from CSV or custom Zipfian/Pareto generators |
| Identical static JSON body | Template Correlation: Dynamic ElFileBody populated from feeder + prior response saveAs() |
Phase 3: Implement in Gatling¶
public class CheckoutLoadSimulation extends Simulation {
HttpProtocolBuilder httpProtocol = http
.baseUrl("https://staging-api.example.com")
.acceptHeader("application/json")
.contentTypeHeader("application/json")
.shareConnections(); // (1)!
FeederBuilder<String> userFeeder = csv("data/users.csv").circular(); // (2)!
ScenarioBuilder shopper = scenario("Realistic Shopper")
.feed(userFeeder)
.exec(http("Browse Catalog")
.get("/products?category=#{category}")
.check(status().is(200), jsonPath("$[0].id").saveAs("productId"))) // (3)!
.pause(Duration.ofSeconds(3), Duration.ofSeconds(8)) // (4)!
.exec(http("Create Order")
.post("/orders")
.body(ElFileBody("bodies/order.json"))
.check(status().is(201)));
{
setUp(
shopper.injectOpen(
rampUsersPerSec(5).to(100).during(120), // (5)!
constantUsersPerSec(100).during(600) // (6)!
)
)
.protocols(httpProtocol)
.assertions(
global().responseTime().percentile(95.0).lt(500),
global().responseTime().percentile(99.0).lt(1000),
global().successfulRequests().percent().gte(99.5)
);
}
}
- Shares connection pools across virtual users when testing behind a reverse proxy/API gateway, or omit to give each virtual user dedicated browser-like connections.
.circular()wraps around cleanly so long-running tests never fail with empty feeder errors.- Correlates real IDs dynamically from the catalog response into the user's
Session. - Randomized uniform think time avoids artificial synchronized request spikes.
- 2-minute ramp-up warms JVM JIT, connection pools, and caches gradually.
- 10-minute steady-state measurement window to validate SLO stability.
Phase 4: Pre-Flight Execution Checklist¶
Run through this checklist before launching every staging load test:
Environment & Data Readiness¶
- Staging topology mirrors production architecture (same DB engine, cache tier, replica ratios)
- Database seeded to production-representative row counts (Volume parity)
- Application log level set to
INFO/WARN(neverDEBUG—synchronous debug logging destroys disk I/O!) - APM tracing (Datadog / OpenTelemetry) and host metrics active with run-specific tags (
test_run:rc-2026-11)
Load Generator & Team Readiness¶
- 1-User Smoke Test passed with 0% KO errors and verified correlation
- Load generator machine has ample CPU (< 70%), RAM, and file-descriptor limits (
ulimit -n >= 65535) - Ops / SRE channels notified so synthetic load alerts don't page on-call unexpectedly
- Abort criteria defined (e.g., abort immediately if error rate > 15% or staging DB affects shared tenants)
Phase 5: Execute Systematically (With Cool-Downs)¶
Never run heavy load profiles back-to-back without letting queues drain, TCP TIME_WAIT sockets clear, and GC compact memory:
- Smoke Run (
1 user, 1 min) $\rightarrow$ Verify 100% functional checks pass. - Baseline Constant Load (
1x expected peak, 15–30 min) $\rightarrow$ Record baseline SLO compliance. - Cool-Down Window (5–10 min) $\rightarrow$ Confirm memory, connections, and CPU return to idle baseline.
- Step / Ramp Capacity Run (
0.5x → 1x → 1.5x → 2x → 3x, 5 min per step) $\rightarrow$ Locate the queuing knee and first bottleneck. - Spike & Soak Runs $\rightarrow$ Validate burst recovery and overnight leak stability.
Phase 6–8: Analyze, Optimize & Re-Verify¶
Symptom-to-Cause Triage Matrix¶
| Gatling Observation | Server Telemetry Clue | Probable Root Cause & Action |
|---|---|---|
| Throughput plateaus flat while latency climbs sharply | DB pool active = max, API CPU < 30% |
Pool / Lock Serialization — optimize slow SQL, add index, tune pool size |
| Latency climbs linearly with RPS until CPU hits 95%+ | High %usr CPU in flame graph |
Compute Bound — optimize hot serialization/crypto paths, cache computed results, scale horizontally |
P50 & P95 stay fast (60 ms), but P99 spikes to 2,500 ms periodically |
Sawtooth heap drops + GC pause logs | JVM GC / Checkpoint Stalls — reduce object allocation rate, switch to G1GC tuned pause target or Generational ZGC |
Sudden jump in ConnectException / Cannot assign requested address |
Load generator TIME_WAIT > 28k |
Generator Ephemeral Port Exhaustion — enable HTTP Keep-Alive, widen ip_local_port_range, distribute generators |
Verify Every Optimization With an Identical Replay
Never ship an optimization on intuition alone. Re-run the exact same Gatling simulation, feeder seed, and injection profile and compare before/after P50, P95, P99, throughput, and CPU side-by-side.
Knowledge Check¶
Why must application logging never be left at DEBUG level during a high-throughput performance test?
When estimating peak RPS from a forecast of 8,640,000 requests per day, why is 8,640,000 / 86,400 = 100 RPS dangerously low as a load test target?
Next Steps¶
- Read next: Load Injection Patterns — Implement Constant, Ramp, Step, Spike, and Closed profiles in Gatling.