Distributed Testing

1 machine: 10,000 concurrent users max
Need: 50,000 concurrent users
Solution: Distribute across 5 machines

┌─────────────────────────────────────────┐
│     Gatling Enterprise Controller       │
│  (Coordinates and aggregates results)   │
└─────────────────────────────────────────┘
         ↙          ↓          ↖
    ┌────────┐ ┌────────┐ ┌────────┐
    │Agent 1 │ │Agent 2 │ │Agent 3 │
    │3,000   │ │3,000   │ │3,000   │
    │users   │ │users   │ │users   │
    │10Gbps  │ │10Gbps  │ │10Gbps  │
    └────────┘ └────────┘ └────────┘
        ↓          ↓          ↓
    ┌──────────────────────────────┐
    │   Target System Under Test   │
    │   (Receives 30,000 users)    │
    └──────────────────────────────┘

✓ Managed platform
✓ Automatic agent coordination
✓ Built-in result aggregation
✓ Visual reporting
✓ Compliance features
✗ Requires subscription

✓ Free
✓ Full control
✗ Manual coordination
✗ Manual result aggregation
✗ More operational overhead

# Machine 1, 2, 3 (Ubuntu servers)
curl -X PUT -d -u admin:admin http://localhost:8080/gatling/data/simulation \
     -H "Content-Type: application/json" \
     -d @simulation.json

// Reference simulation on agents
// Each agent runs same simulation with different user offset

public class Sim_DistributedLoad extends Simulation {

    // Get agent ID (0, 1, 2, etc.)
    String agentId = System.getProperty("gatling.agentId", "0");

    // Distribute users across agents
    // Agent 0: users 0-9999
    // Agent 1: users 10000-19999
    // Agent 2: users 20000-29999

    int userOffset = Integer.parseInt(agentId) * 10000;

    ScenarioBuilder scenario = scenario("Distributed Load")
        .feed(userFeeder.offset(userOffset))
        .exec(http("Request").get("/api"));
}

# Agent 1
mvn gatling:test \
  -Dgatling.simulationClass=Sim_DistributedLoad \
  -Dgatling.agentId=0

# Agent 2
mvn gatling:test \
  -Dgatling.simulationClass=Sim_DistributedLoad \
  -Dgatling.agentId=1

# Agent 3
mvn gatling:test \
  -Dgatling.simulationClass=Sim_DistributedLoad \
  -Dgatling.agentId=2

# Collect results from each agent
# Manually merge CSV files:

cat agent1/results.csv agent2/results.csv agent3/results.csv > combined.csv

# Calculate aggregated metrics
# Total requests = sum of all agents
# P95 latency = p95 of combined data

Agent 1: starts at 10:00:00
Agent 2: starts at 10:00:05  ← 5 second delay
Agent 3: starts at 10:00:10  ← 10 second delay

Result: Load is staggered, not simultaneous

// Use barrier to wait for all agents
Barrier barrier = new Barrier(3);  // 3 agents

// All agents wait at barrier
barrier.await();  // Blocks until all 3 reach this point

// Then start simultaneously
setUp(scenario.injectOpen(...))

Agent 1: 10,000 requests, p95=450ms, errors=2
Agent 2: 10,000 requests, p95=480ms, errors=3
Agent 3: 10,000 requests, p95=520ms, errors=1

Total: 30,000 requests
P95: ((450*10000 + 480*10000 + 520*10000) / 30000) = 483ms
Errors: 2 + 3 + 1 = 6 (0.02% error rate)

Per user: ~1MB data per second
Per machine (1000 concurrent users): ~1Gbps
Per machine (10,000 concurrent users): ~10Gbps

3 machines with 10,000 users each:
├─ Each machine: 10Gbps
├─ Total to target: 30Gbps
└─ Network: Must have ≥30Gbps capacity

Datacenter network: Typically 10Gbps per server
3 servers: 30Gbps total available
3 servers hitting target: 30Gbps required
Result: Perfect fit (but no headroom)

Better: Use 5 machines with 6,000 users each
├─ Per machine: 6Gbps
├─ Total: 30Gbps (same)
└─ Headroom: Yes, less contention

Agents and target on same network
└─ Minimize latency

Avoid routing through internet
└─ Variable latency ruins test

# All machines must have synchronized clocks
ntpdate -u ntp.ubuntu.com  # Sync to NTP

# Verify
timedatectl  # Check clock is synchronized

Per agent machine:
├─ CPU: 16 cores (for 10,000 users)
├─ RAM: 32GB (for 10,000 users)
├─ Network: 10Gbps+ NIC
└─ Storage: Fast SSD for logging

Monitor each agent during test:
├─ CPU: Should not exceed 80%
├─ Memory: Should not exceed 80%
├─ Network: Should not exceed 90%

If exceeded: Add more agents, reduce per-agent users

Agent 1: 10,000 requests
Agent 2: 8,000 requests
Agent 3: 9,000 requests

Problem: Agents started at different times
Solution: Add synchronization barrier

Error: OutOfMemoryError
Solution: Reduce users per agent or increase JVM heap

Observation: Network at 100%, latency high
Solution: Add more agents with fewer users each

Pros:
✓ Automatic scaling (0-100,000+ users)
✓ Managed cloud infrastructure
✓ Built-in reporting
✓ Real-time dashboards
✓ Compliance features

Cons:
✗ Cost ($$$)
✗ Less control
✗ Vendor lock-in

Use when: Load >50,000 users, team size >5, budget available