Skip to content

Load Testing Methodology

Repeatable, trustworthy performance results come from a scientific, eight-phase engineering methodology—never from firing ad-hoc traffic at an unprepared environment.

flowchart LR
    P1["1. Plan<br/>SLOs & Critical Paths"] --> P2["2. Model<br/>Traffic Mix & Data"]
    P2 --> P3["3. Implement<br/>Gatling Simulation"]
    P3 --> P4["4. Smoke<br/>1-User Validation"]
    P4 --> P5["5. Execute<br/>Warm-Up + Target Profile"]
    P5 --> P6["6. Analyze<br/>Gatling + APM Traces"]
    P6 --> P7["7. Optimize<br/>Single-Variable Fix"]
    P7 --> P8["8. Re-Verify<br/>Regression Gate"]
    P8 -.->|"Next bottleneck"| P5

Phase 1: Plan & Quantify Objectives

Define Measurable Acceptance Criteria

Before writing simulation code, align Product, SRE, and Engineering on quantitative pass/fail gates:

Service Path Peak Target Load P95 SLO P99 SLO Max Error Rate
GET /products (Catalog Browse) 3,500 RPS < 150 ms < 350 ms < 0.05%
POST /cart (Cart Mutation) 800 RPS < 250 ms < 500 ms < 0.10%
POST /checkout (Order Commit) 200 RPS < 500 ms < 1,000 ms < 0.05%

Estimate Target Load from Production Telemetry

  1. Historical Peak: Query last peak event RPS (e.g., Black Friday, Monday 9 AM login surge).
  2. Peak-to-Average Ratio: Web traffic peaks are typically 3x–8x daily average RPS! Never divide daily requests by 86,400 without applying a peak multiplier.
  3. Safety Headroom: Validate up to 1.5x–2x forecasted peak so unexpected growth or a single AZ failure stays below the queuing knee.

Phase 2: Design Realistic Workloads & Data

Avoid Synthetic Monoculture

Anti-Pattern Production-Realistic Design
100% of users hit POST /checkout with zero pauses 80/20 Funnel: 70% browse/search, 20% add to cart, 10% checkout with randomized pause(3, 12)
Every iteration queries userId = 1 Parameterized Feeders: 50,000+ distinct IDs from CSV or custom Zipfian/Pareto generators
Identical static JSON body Template Correlation: Dynamic ElFileBody populated from feeder + prior response saveAs()

Phase 3: Implement in Gatling

public class CheckoutLoadSimulation extends Simulation {

    HttpProtocolBuilder httpProtocol = http
        .baseUrl("https://staging-api.example.com")
        .acceptHeader("application/json")
        .contentTypeHeader("application/json")
        .shareConnections(); // (1)!

    FeederBuilder<String> userFeeder = csv("data/users.csv").circular(); // (2)!

    ScenarioBuilder shopper = scenario("Realistic Shopper")
        .feed(userFeeder)
        .exec(http("Browse Catalog")
            .get("/products?category=#{category}")
            .check(status().is(200), jsonPath("$[0].id").saveAs("productId"))) // (3)!
        .pause(Duration.ofSeconds(3), Duration.ofSeconds(8)) // (4)!
        .exec(http("Create Order")
            .post("/orders")
            .body(ElFileBody("bodies/order.json"))
            .check(status().is(201)));

    {
        setUp(
            shopper.injectOpen(
                rampUsersPerSec(5).to(100).during(120),      // (5)!
                constantUsersPerSec(100).during(600)         // (6)!
            )
        )
        .protocols(httpProtocol)
        .assertions(
            global().responseTime().percentile(95.0).lt(500),
            global().responseTime().percentile(99.0).lt(1000),
            global().successfulRequests().percent().gte(99.5)
        );
    }
}
  1. Shares connection pools across virtual users when testing behind a reverse proxy/API gateway, or omit to give each virtual user dedicated browser-like connections.
  2. .circular() wraps around cleanly so long-running tests never fail with empty feeder errors.
  3. Correlates real IDs dynamically from the catalog response into the user's Session.
  4. Randomized uniform think time avoids artificial synchronized request spikes.
  5. 2-minute ramp-up warms JVM JIT, connection pools, and caches gradually.
  6. 10-minute steady-state measurement window to validate SLO stability.

Phase 4: Pre-Flight Execution Checklist

Run through this checklist before launching every staging load test:

Environment & Data Readiness

  • Staging topology mirrors production architecture (same DB engine, cache tier, replica ratios)
  • Database seeded to production-representative row counts (Volume parity)
  • Application log level set to INFO / WARN (never DEBUG—synchronous debug logging destroys disk I/O!)
  • APM tracing (Datadog / OpenTelemetry) and host metrics active with run-specific tags (test_run:rc-2026-11)

Load Generator & Team Readiness

  • 1-User Smoke Test passed with 0% KO errors and verified correlation
  • Load generator machine has ample CPU (< 70%), RAM, and file-descriptor limits (ulimit -n >= 65535)
  • Ops / SRE channels notified so synthetic load alerts don't page on-call unexpectedly
  • Abort criteria defined (e.g., abort immediately if error rate > 15% or staging DB affects shared tenants)

Phase 5: Execute Systematically (With Cool-Downs)

Never run heavy load profiles back-to-back without letting queues drain, TCP TIME_WAIT sockets clear, and GC compact memory:

  1. Smoke Run (1 user, 1 min) $\rightarrow$ Verify 100% functional checks pass.
  2. Baseline Constant Load (1x expected peak, 15–30 min) $\rightarrow$ Record baseline SLO compliance.
  3. Cool-Down Window (5–10 min) $\rightarrow$ Confirm memory, connections, and CPU return to idle baseline.
  4. Step / Ramp Capacity Run (0.5x → 1x → 1.5x → 2x → 3x, 5 min per step) $\rightarrow$ Locate the queuing knee and first bottleneck.
  5. Spike & Soak Runs $\rightarrow$ Validate burst recovery and overnight leak stability.

Phase 6–8: Analyze, Optimize & Re-Verify

Symptom-to-Cause Triage Matrix

Gatling Observation Server Telemetry Clue Probable Root Cause & Action
Throughput plateaus flat while latency climbs sharply DB pool active = max, API CPU < 30% Pool / Lock Serialization — optimize slow SQL, add index, tune pool size
Latency climbs linearly with RPS until CPU hits 95%+ High %usr CPU in flame graph Compute Bound — optimize hot serialization/crypto paths, cache computed results, scale horizontally
P50 & P95 stay fast (60 ms), but P99 spikes to 2,500 ms periodically Sawtooth heap drops + GC pause logs JVM GC / Checkpoint Stalls — reduce object allocation rate, switch to G1GC tuned pause target or Generational ZGC
Sudden jump in ConnectException / Cannot assign requested address Load generator TIME_WAIT > 28k Generator Ephemeral Port Exhaustion — enable HTTP Keep-Alive, widen ip_local_port_range, distribute generators

Verify Every Optimization With an Identical Replay

Never ship an optimization on intuition alone. Re-run the exact same Gatling simulation, feeder seed, and injection profile and compare before/after P50, P95, P99, throughput, and CPU side-by-side.


Knowledge Check

#

Why must application logging never be left at DEBUG level during a high-throughput performance test?

#

When estimating peak RPS from a forecast of 8,640,000 requests per day, why is 8,640,000 / 86,400 = 100 RPS dangerously low as a load test target?


Next Steps