Common Pitfalls & Shift-Left CI Gates¶
Even experienced engineering teams invalidate weeks of load testing by falling into subtle measurement traps. Use this guide to audit your test suite and wire automated Shift-Left performance regression gates into CI/CD.
Top 10 Performance Testing Pitfalls¶
1. Zero Think Time (Unrealistic Hammering)¶
scenario("Realistic Shopper")
.feed(productFeeder)
.exec(http("Catalog").get("/api/products"))
.pause(Duration.ofSeconds(5), Duration.ofSeconds(15))
.exec(http("Detail").get("/api/products/#{productId}"))
.pause(Duration.ofSeconds(8), Duration.ofSeconds(20))
.exec(http("Cart").post("/api/cart"))
.pause(Duration.ofSeconds(10), Duration.ofSeconds(25))
.exec(http("Checkout").post("/api/checkout"));
Omitting pauses distorts connection hold times, session memory footprint, and read-to-write concurrency ratios. (When testing pure stateless microservice-to-microservice RPS limits, omit user think time intentionally and control arrival rate via injectOpen.)
2. Saturating the Load Generator¶
If you drive 15,000 users from a laptop at 100% CPU or exhaust OS ephemeral ports, Gatling stalls waiting for local CPU/sockets—inflating reported latency while the target API sits idle at 20% CPU!
Rule of Thumb: Monitor the Tester!
Keep load generator CPU below 70%, heap utilization below 75%, and raise open file descriptors (ulimit -n 65535). Scale horizontally across multiple load generators when needed (see Distributed Testing).
3. Skipping the Warm-Up Phase¶
Cold JVMs interpret bytecode slowly before C1/C2 JIT compilation kicks in, connection pools start empty, and Redis/OS page caches are cold. Measuring a 2-minute cold burst blends one-time startup penalties into steady-state SLAs. Always ramp or warm up for 2–5 minutes before your steady-state window.
4. The Single-Record Cache Illusion¶
Testing GET /api/users/1 500,000 times benchmarks L1 CPU cache / Redis, never touching disk B-trees or realistic lock distribution. Always parameterize requests with diverse feeders (csv(), jsonFile(), or custom Iterator<Map<String, Object>>).
5. HTTP 200 "Soft Error" Blindness¶
Many GraphQL APIs, legacy gateways, or fallback handlers return HTTP 200 OK containing {"error": "Database timeout"} or an empty [] list! Relying solely on Gatling's default 2xx check marks 100% of broken responses as OK—always assert payload structure:
.exec(http("Fetch Order")
.get("/api/orders/#{orderId}")
.check(
status().is(200),
jsonPath("$.orderId").isEL("#{orderId}"), // Proves valid business payload!
jsonPath("$.error").notExists()
))
6. Ignoring Downstream Mocks & Third-Party Sandboxes¶
Pointing a 5,000 RPS staging test at Stripe's or Twilio's free shared sandbox will immediately trigger rate-limiting (HTTP 429) and get your IP blocked, while mocking downstream calls with 0 ms in-memory stubs hides thread-pool starvation!
Fix: Stub external third-party APIs (e.g., WireMock) configured with realistic latency distributions (lognormal 120 ms median + 1% 1,500 ms tail + 0.2% 503 errors).
7. Testing Happy Path Only¶
Real traffic includes invalid tokens (401), validation failures (400), cache misses (404), and occasional downstream timeouts (504). Error paths often trigger expensive exception stack-trace generation (Throwable.fillInStackTrace()) or uncached DB lookups that degrade CPU under load! Include a realistic 1–5% negative-path scenario in your test mix.
8. Uncoordinated Production Load Testing¶
Never point synthetic write load at production without explicit executive/SRE sign-off, tenant isolation, synthetic traffic headers, payment-gateway bypasses, and an instant kill switch. Polluting production analytics, billing ledgers, or inventory tables is an avoidable self-inflicted outage.
9. Tracking Mean Latency or Averaging Percentiles¶
Mean latency hides tail outages, and averaging P99s across pods is mathematically invalid. Always enforce assertions on P95, P99, and P99.9 computed from full response histograms.
10. Manual "Once a Year" Testing Instead of Shift-Left CI¶
Waiting until the week before Black Friday to run Gatling means discovering architectural bottlenecks across 400 merged commits when it is too late to redesign.
Shift-Left Performance Gates in CI/CD¶
Automate two tiers of Gatling execution in GitHub Actions / GitLab CI:
flowchart LR
PR["Pull Request / Merge"] --> Smoke["Tier 1: Fast CI Gate (2–3 min)<br/>Functional checks + Smoke SLO assertions<br/>Fails PR immediately on regression"]
Smoke --> Nightly["Tier 2: Scheduled Staging Gate (Nightly)<br/>30m Step & Constant Load + APM Tagging<br/>Tracks P95/P99 trend across releases"]
Production-Grade GitHub Actions Gate¶
Because gatling-maven-plugin has <failOnError>true</failOnError> enabled in pom.xml, any failed .assertions(...) block automatically returns a non-zero exit code and fails the CI pipeline—no brittle HTML grep hacks required!
name: Shift-Left Performance Gate
on:
pull_request:
branches: [ main ]
schedule:
- cron: '0 2 * * *' # Nightly regression suite at 02:00 UTC
jobs:
gatling-slo-gate:
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Set up Java 21
uses: actions/setup-java@v4
with:
java-version: '21'
distribution: 'temurin'
cache: maven
- name: Run Gatling Simulation with SLO Assertions
run: |
mvn gatling:test \
-Dgatling.simulationClass=io.learn.gatling.simulations.http.Sim03_HttpChecks \
--no-transfer-progress
- name: Upload Gatling HTML Report Artifact
if: always()
uses: actions/upload-artifact@v4
with:
name: gatling-report-${{ github.run_number }}
path: target/gatling/
retention-days: 14
Summary Checklist¶
| Do This | Avoid This |
|---|---|
Model Open arrivals (injectOpen) for public traffic |
Using Closed loops that self-throttle during stalls |
| Parameterize data with CSV / custom feeders | Hitting /users/1 100,000 times from RAM cache |
| Warm up JIT & pools before steady-state measurement | Judging steady-state SLAs from a 30-second cold burst |
Assert on payload fields (jsonPath) + status codes |
Trusting bare HTTP 200 on GraphQL / fallback APIs |
| Stub third-party APIs with realistic latency distributions | Hammering vendor sandboxes or using 0 ms mocks |
Enforce .assertions(...) automatically in CI |
Running manual load tests once a year before launch |
Knowledge Check¶
Why is mocking an external payment API with a 0 ms instant response dangerous during staging load tests?
How should you fail a Maven CI pipeline automatically when P95 latency exceeds 500 ms or success rate drops below 99.5%?
Next Steps¶
- Continue to Gatling Engine: Gatling Architecture — Explore how Gatling's non-blocking Netty event-loop engine simulates tens of thousands of users on a single JVM.