Skip to content

HTTP vs. Kafka Load Testing Patterns

Modern distributed architectures combine synchronous request-response APIs (HTTP/gRPC) with asynchronous event-streaming backbones (Apache Kafka). Load testing both requires understanding how their latency boundaries, failure modes, and success criteria differ fundamentally.


Architectural Comparison

sequenceDiagram
    participant C as Gatling Client
    participant API as Order HTTP Service
    participant K as Kafka Broker
    participant W as Downstream Consumer (Inventory)

    rect rgb(235, 245, 255)
    Note over C,API: Pattern A — Synchronous HTTP (End-to-End Request/Response)
    C->>API: POST /api/orders
    API-->>C: 201 Created (Client measures full round-trip: 145ms)
    end

    rect rgb(245, 240, 235)
    Note over C,W: Pattern B — Asynchronous Kafka (Decoupled Ack vs. Consumer Lag)
    C->>K: Produce OrderCreated (acks=all)
    K-->>C: RecordMetadata Ack (Producer measures broker ack: 8ms)
    K->>W: Poll Batch
    Note over W: Process & Commit Offset (Monitored via Consumer Lag!)
    end
Dimension HTTP Simulations (Labs 1–5) Kafka Simulations (Labs 6–7)
Communication Model Synchronous Request-Response Asynchronous Publish-Subscribe
Gatling Support Native HttpDsl (http("...").get(...)) Custom exec(session -> ...) invoking KafkaProducer
What Gatling Latency Shows Full client-to-server round-trip time Producer-to-Broker acks latency (NOT downstream consumer processing!)
Primary Overload Symptom Rising P95/P99 HTTP latency, 503/504 timeouts Growing Consumer Group Lag (records-lag-max), broker request queue saturation
Routing & Ordering Stateless URL / Load-balancer round-robin Message Key hashing (murmur2(key) % numPartitions) preserves per-entity ordering
Sync vs. Async Send Non-blocking Netty I/O under the hood Sync send().get() (Lab 6) vs. Async send(record, callback) batching (Lab 7)

Sync vs. Async Kafka Producer Patterns

In Lab 6 (Sim06_BasicKafkaProducer) and Lab 7 (Sim07_KafkaWithFeeders), you will compare both producer patterns:

Waits for broker acknowledgment on every single message before advancing:

.exec(session -> {
    try {
        producer.send(new ProducerRecord<>(topic, key, payload)).get();
        return session.markAsSucceeded();
    } catch (Exception e) {
        return session.markAsFailed();
    }
})
- Pros: Simple mental model; directly measures per-record broker ack time. - Cons: Blocks caller thread during network RTT; defeats Kafka batching (linger.ms), capping throughput per thread.

Leverages Kafka's native batching (batch.size + linger.ms) for massive throughput:

.exec(session -> {
    producer.send(new ProducerRecord<>(topic, key, payload), (metadata, ex) -> {
        if (ex != null) {
            logger.error("Kafka send failed", ex);
        }
    });
    return session;
})
- Pros: Matches real high-throughput production producers (tens of thousands of events/sec). - Cons: Must monitor broker metrics, callback error counters, and downstream consumer lag in APM!

Producer Green != System Healthy!

A Kafka broker can effortlessly ingest 50,000 messages/second in 5 ms while your downstream Spring Kafka consumer only processes 1,000 messages/second! If you only check Gatling's producer report, the test looks 100% green while consumer lag balloons by 49,000 messages every second. Always monitor kafka.consumer_group.lag alongside Kafka producer tests!


Knowledge Check

#

During a Kafka producer load test at 10,000 messages/sec, Gatling reports 0% errors and P99 send latency of 12 ms, yet customers complain order emails arrive 45 minutes late. Why?


Next Steps

  • Read next: Scenarios & Feeders — Parameterize HTTP and Kafka user journeys with CSV, JSON, and custom feeders in Java DSL.