04 · Consumer Groups & Offsets

Consumer Groups Recap

A consumer group allows multiple application instances to cooperate in consuming a topic:

events-topic (3 partitions)
┌─────────────┬─────────────┬─────────────┐
│ Partition 0 │ Partition 1 │ Partition 2 │
└──────┬──────┴──────┬──────┴──────┬──────┘
       │             │             │
   Instance A    Instance B    Instance C
   (manual-ack-group)

Rules:

• Each partition → at most one consumer in the group
• Excess consumers are idle (standby for failover)
• Each group tracks its own committed offsets independently
• Adding a new consumer group = full replay from the auto.offset.reset point

The Three Groups in This Project

graph TD
    ET[events-topic<br/>3 partitions] -->|manual-ack-group| MAC[ManualAckConsumer]
    DLT[events-topic.DLT<br/>1 partition] -->|dlt-consumer-group| DLC[DeadLetterConsumer]
    AET[avro-events-topic<br/>3 partitions] -->|avro-consumer-group| AEC[AvroEventConsumer]

auto.offset.reset

When a consumer group has no committed offset (new group, or offsets expired), Kafka uses auto.offset.reset to decide where to start reading:

props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest");

Offset Commit Strategies

graph LR
    A[Message received<br/>offset=42] --> B{Process}
    B -->|success| C[ack.acknowledge<br/>commit offset 43]
    B -->|fail| D[throw exception<br/>offset stays at 42<br/>→ retry]
    D -->|after max retries| E[DeadLetterPublishingRecoverer<br/>→ DLT, commit offset 43]

enable.auto.commit = false (this project)

props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);

Spring controls the commit. Offset is committed only when ack.acknowledge() is called (with MANUAL_IMMEDIATE). This gives:

• At-least-once delivery — a message may be re-processed if the app crashes after processing but before commit
• No silent data loss — offset never advances past unprocessed messages

enable.auto.commit = true (not used here)

Kafka commits offsets at a fixed interval (auto.commit.interval.ms). Simpler but:

• Messages may be committed before they're successfully processed → data loss on crash
• Messages may be re-processed if the app crashes between a commit interval → duplicates possible

Rebalancing

A rebalance occurs when the group membership changes:

• A new consumer joins (scale-out)
• A consumer leaves or crashes
• Topics/partitions are added

During a rebalance: 1. All consumers in the group stop consuming (pause) 2. The Group Coordinator (a Kafka broker) reassigns partitions 3. Consumers resume from their committed offsets on the newly assigned partitions

sequenceDiagram
    participant GC as Group Coordinator
    participant C1 as Consumer 1
    participant C2 as Consumer 2 (new)

    Note over C1: consuming P0, P1, P2
    C2->>GC: JoinGroup request
    GC->>C1: Stop consuming (rebalance triggered)
    GC->>GC: Assign: C1→P0,P1 / C2→P2
    GC->>C1: Resume on P0, P1
    GC->>C2: Start on P2 from committed offset

Lag — the Key Observability Metric

Consumer lag = (latest offset on partition) − (consumer's committed offset)

Partition 0: latest=100, committed=98 → lag=2
Partition 1: latest=50,  committed=50 → lag=0
Partition 2: latest=75,  committed=70 → lag=5

Total lag for manual-ack-group = 7

Zero lag = consumer is fully caught up. High/growing lag = consumer is falling behind — scale out or investigate.

View in Kafka-UI → Consumers → manual-ack-group.

Offset Storage

Before Kafka 0.9, offsets were stored in ZooKeeper. Now they're stored in an internal Kafka topic:

__consumer_offsets (50 partitions)

This topic records (group_id, topic, partition) → committed_offset for every consumer group. You don't need to interact with it directly — ack.acknowledge() writes to it transparently.

Key Takeaways

Up Next

➡️ 05 · Error Handling & DLT

Hands-on now? → Lab 03 · Consumer Groups