Skip to content

Common Pitfalls & Shift-Left CI Gates

Even experienced engineering teams invalidate weeks of load testing by falling into subtle measurement traps. Use this guide to audit your test suite and wire automated Shift-Left performance regression gates into CI/CD.


Top 10 Performance Testing Pitfalls

1. Zero Think Time (Unrealistic Hammering)

scenario("Unrealistic Bot")
    .exec(http("Catalog").get("/api/products"))
    .exec(http("Detail").get("/api/products/42"))
    .exec(http("Cart").post("/api/cart"))
    .exec(http("Checkout").post("/api/checkout")); // Fires in < 200ms!
scenario("Realistic Shopper")
    .feed(productFeeder)
    .exec(http("Catalog").get("/api/products"))
    .pause(Duration.ofSeconds(5), Duration.ofSeconds(15))
    .exec(http("Detail").get("/api/products/#{productId}"))
    .pause(Duration.ofSeconds(8), Duration.ofSeconds(20))
    .exec(http("Cart").post("/api/cart"))
    .pause(Duration.ofSeconds(10), Duration.ofSeconds(25))
    .exec(http("Checkout").post("/api/checkout"));

Omitting pauses distorts connection hold times, session memory footprint, and read-to-write concurrency ratios. (When testing pure stateless microservice-to-microservice RPS limits, omit user think time intentionally and control arrival rate via injectOpen.)


2. Saturating the Load Generator

If you drive 15,000 users from a laptop at 100% CPU or exhaust OS ephemeral ports, Gatling stalls waiting for local CPU/sockets—inflating reported latency while the target API sits idle at 20% CPU!

Rule of Thumb: Monitor the Tester!

Keep load generator CPU below 70%, heap utilization below 75%, and raise open file descriptors (ulimit -n 65535). Scale horizontally across multiple load generators when needed (see Distributed Testing).


3. Skipping the Warm-Up Phase

Cold JVMs interpret bytecode slowly before C1/C2 JIT compilation kicks in, connection pools start empty, and Redis/OS page caches are cold. Measuring a 2-minute cold burst blends one-time startup penalties into steady-state SLAs. Always ramp or warm up for 2–5 minutes before your steady-state window.


4. The Single-Record Cache Illusion

Testing GET /api/users/1 500,000 times benchmarks L1 CPU cache / Redis, never touching disk B-trees or realistic lock distribution. Always parameterize requests with diverse feeders (csv(), jsonFile(), or custom Iterator<Map<String, Object>>).


5. HTTP 200 "Soft Error" Blindness

Many GraphQL APIs, legacy gateways, or fallback handlers return HTTP 200 OK containing {"error": "Database timeout"} or an empty [] list! Relying solely on Gatling's default 2xx check marks 100% of broken responses as OK—always assert payload structure:

.exec(http("Fetch Order")
    .get("/api/orders/#{orderId}")
    .check(
        status().is(200),
        jsonPath("$.orderId").isEL("#{orderId}"), // Proves valid business payload!
        jsonPath("$.error").notExists()
    ))

6. Ignoring Downstream Mocks & Third-Party Sandboxes

Pointing a 5,000 RPS staging test at Stripe's or Twilio's free shared sandbox will immediately trigger rate-limiting (HTTP 429) and get your IP blocked, while mocking downstream calls with 0 ms in-memory stubs hides thread-pool starvation!

Fix: Stub external third-party APIs (e.g., WireMock) configured with realistic latency distributions (lognormal 120 ms median + 1% 1,500 ms tail + 0.2% 503 errors).


7. Testing Happy Path Only

Real traffic includes invalid tokens (401), validation failures (400), cache misses (404), and occasional downstream timeouts (504). Error paths often trigger expensive exception stack-trace generation (Throwable.fillInStackTrace()) or uncached DB lookups that degrade CPU under load! Include a realistic 1–5% negative-path scenario in your test mix.


8. Uncoordinated Production Load Testing

Never point synthetic write load at production without explicit executive/SRE sign-off, tenant isolation, synthetic traffic headers, payment-gateway bypasses, and an instant kill switch. Polluting production analytics, billing ledgers, or inventory tables is an avoidable self-inflicted outage.


9. Tracking Mean Latency or Averaging Percentiles

Mean latency hides tail outages, and averaging P99s across pods is mathematically invalid. Always enforce assertions on P95, P99, and P99.9 computed from full response histograms.


10. Manual "Once a Year" Testing Instead of Shift-Left CI

Waiting until the week before Black Friday to run Gatling means discovering architectural bottlenecks across 400 merged commits when it is too late to redesign.


Shift-Left Performance Gates in CI/CD

Automate two tiers of Gatling execution in GitHub Actions / GitLab CI:

flowchart LR
    PR["Pull Request / Merge"] --> Smoke["Tier 1: Fast CI Gate (2–3 min)<br/>Functional checks + Smoke SLO assertions<br/>Fails PR immediately on regression"]
    Smoke --> Nightly["Tier 2: Scheduled Staging Gate (Nightly)<br/>30m Step & Constant Load + APM Tagging<br/>Tracks P95/P99 trend across releases"]

Production-Grade GitHub Actions Gate

Because gatling-maven-plugin has <failOnError>true</failOnError> enabled in pom.xml, any failed .assertions(...) block automatically returns a non-zero exit code and fails the CI pipeline—no brittle HTML grep hacks required!

name: Shift-Left Performance Gate

on:
  pull_request:
    branches: [ main ]
  schedule:
    - cron: '0 2 * * *' # Nightly regression suite at 02:00 UTC

jobs:
  gatling-slo-gate:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Set up Java 21
        uses: actions/setup-java@v4
        with:
          java-version: '21'
          distribution: 'temurin'
          cache: maven

      - name: Run Gatling Simulation with SLO Assertions
        run: |
          mvn gatling:test \
            -Dgatling.simulationClass=io.learn.gatling.simulations.http.Sim03_HttpChecks \
            --no-transfer-progress

      - name: Upload Gatling HTML Report Artifact
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: gatling-report-${{ github.run_number }}
          path: target/gatling/
          retention-days: 14

Summary Checklist

Do This Avoid This
Model Open arrivals (injectOpen) for public traffic Using Closed loops that self-throttle during stalls
Parameterize data with CSV / custom feeders Hitting /users/1 100,000 times from RAM cache
Warm up JIT & pools before steady-state measurement Judging steady-state SLAs from a 30-second cold burst
Assert on payload fields (jsonPath) + status codes Trusting bare HTTP 200 on GraphQL / fallback APIs
Stub third-party APIs with realistic latency distributions Hammering vendor sandboxes or using 0 ms mocks
Enforce .assertions(...) automatically in CI Running manual load tests once a year before launch

Knowledge Check

#

Why is mocking an external payment API with a 0 ms instant response dangerous during staging load tests?

#

How should you fail a Maven CI pipeline automatically when P95 latency exceeds 500 ms or success rate drops below 99.5%?


Next Steps

  • Continue to Gatling Engine: Gatling Architecture — Explore how Gatling's non-blocking Netty event-loop engine simulates tens of thousands of users on a single JVM.