07 · Advanced Patterns

Multi-Factory Configuration

This project runs two serialisation formats (JSON and Avro) side-by-side using separate factories:

graph LR
    subgraph JSON Flow
        JF[ProducerFactory<String,Object>] --> JT[KafkaTemplate<String,Object>]
        JCF[ConsumerFactory<String,Event>] --> JLF[kafkaListenerContainerFactory]
    end
    subgraph Avro Flow
        AF[ProducerFactory<String,SpecificRecord>] --> AT[avroKafkaTemplate]
        ACF[ConsumerFactory<String,SpecificRecord>] --> ALF[avroKafkaListenerContainerFactory]
    end

The containerFactory attribute on @KafkaListener selects which factory to use:

// Uses the DEFAULT factory (kafkaListenerContainerFactory)
@KafkaListener(topics = "${kafka.topics.main}", groupId = "manual-ack-group")
public void consume(@Payload Event event, ...) { }

// Explicitly names the Avro factory
@KafkaListener(
    topics = "${kafka.topics.avro}",
    groupId = "avro-consumer-group",
    containerFactory = "avroKafkaListenerContainerFactory"  // ← key
)
public void consume(@Payload AvroEvent event, ...) { }

ConcurrentKafkaListenerContainerFactory — Concurrency

factory.setConcurrency(3); // creates 3 consumer threads per listener

With concurrency=3 and a 3-partition topic, each thread owns one partition:

Thread 1 → Partition 0
Thread 2 → Partition 1
Thread 3 → Partition 2

With concurrency=1 (the project default — not explicitly set), one thread handles all partitions sequentially.

Idempotent Producers

By default, Kafka producers can produce duplicate messages if the broker acknowledges the write but the network drops the ACK before the producer receives it → producer retries → duplicate.

Enable idempotence:

props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
// Automatically sets:
// acks=all, retries=MAX_INT, max.in.flight.requests.per.connection=5

With idempotent producers, Kafka deduplicates retries using a sequence number — each (producer_id, partition, sequence_number) is written exactly once.

Exactly-Once Semantics (EOS)

EOS = exactly-once end-to-end (producer → broker → consumer). Requires:

1. Idempotent producer (above)
2. Transactions on the producer side:

props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "my-app-tx-1");
kafkaTemplate.executeInTransaction(ops -> {
    ops.send(topicA, key, valueA);
    ops.send(topicB, key, valueB);
    return true;
}); // either both messages are written, or neither

1. Consumer isolation.level=read_committed:

   props.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
   // Consumers only see committed transactional messages
   

Partitioning Strategies

Default — Sticky Partitioner (Kafka 2.4+)

When no key is specified, the producer batches messages to the same partition until the batch is full or the linger time expires, then rotates. This maximises throughput.

Key-Based — Murmur2 Hash

partition = murmur2(key.getBytes()) % numPartitions

Same key → same partition → ordering guarantee for related events.

In this project, event.getEventId() is used as the key. Since UUIDs are random, events distribute evenly across partitions.

Custom Partitioner

public class OrderPartitioner implements Partitioner {
    @Override
    public int partition(String topic, Object key, byte[] keyBytes,
                         Object value, byte[] valueBytes, Cluster cluster) {
        // Route VIP orders to partition 0 for priority processing
        if (key.toString().startsWith("VIP-")) return 0;
        return murmur2(keyBytes) % (cluster.partitionCountForTopic(topic) - 1) + 1;
    }
}

Register it:

props.put(ProducerConfig.PARTITIONER_CLASS_CONFIG, OrderPartitioner.class);

When to Add Partitions

Kafka in Microservices Architecture

graph TD
    OCS[Order Service] -->|ORDER_PLACED| K[Kafka]
    K -->|ORDER_PLACED| BS[Billing Service]
    K -->|ORDER_PLACED| FS[Fulfillment Service]
    K -->|ORDER_PLACED| NFS[Notification Service]

    BS -->|PAYMENT_PROCESSED| K
    K -->|PAYMENT_PROCESSED| FS
    K -->|PAYMENT_PROCESSED| NFS

    FS -->|ORDER_SHIPPED| K
    K -->|ORDER_SHIPPED| NFS
    K -->|ORDER_SHIPPED| OCS

Key Design Principles

1. Events, not commands — publish what happened ("ORDER_PLACED"), not instructions ("PROCESS_ORDER"). Any service can react independently.
2. Consumer owns its schema — each service defines what fields it needs. Use a shared schema registry to enforce contracts.
3. Choreography over orchestration — services react to events; no central coordinator needed. This reduces coupling.
4. Bounded retry per service — each service has its own DLT. A failure in Billing doesn't affect Fulfillment.

Monitoring Checklist

Key Takeaways

Congratulations! 🎉

You've completed all 7 theory modules. Head to the Interview Guide to test your knowledge, or revisit labs for hands-on reinforcement.

➡️ Interview Guide