Skip to content

18 · Change Data Capture (CDC) — Stream Database Changes Downstream

Distributed Transactions · Topic 18 of 20


What is CDC?

Change Data Capture tracks row-level changes (INSERT, UPDATE, DELETE) in a database and streams them to downstream consumers in real time.

CDC turns the database into an event stream.


Use Cases

Use Case How CDC Helps
Search index sync Sync Elasticsearch on every DB write
Cache invalidation Invalidate Redis entries when data changes
Audit logging Immutable log of all data changes
Data replication Replicate to another DB or data warehouse
Event-driven pipelines Trigger microservices on data changes

How CDC Works

sequenceDiagram
    App->>Database: INSERT / UPDATE / DELETE
    Database->>WAL: Append change record
    CDC Tool->>WAL: Read changes (tail the log)
    CDC Tool->>Kafka: Publish change events
    Consumers->>Kafka: Read and process events

Log-based CDC, illustrated above, reads the WAL or equivalent committed-change stream. CDC is a broader category that also includes trigger-based capture and polling; those approaches have different overhead and correctness trade-offs.


CDC Tools

Tool Source DBs Target
Debezium PostgreSQL, MySQL, MongoDB, SQL Server Kafka, Kinesis
AWS DMS Most relational DBs S3, Kinesis, RDS
Striim Multi-source Kafka, GCS, BigQuery
Datastream (GCP) PostgreSQL, MySQL, Oracle BigQuery, GCS, Spanner

Cloud Implementations

  • Enable logical replication: wal_level = logical
  • Create replication slot: pg_create_logical_replication_slot('debezium', 'pgoutput')
  • Debezium reads from the slot and publishes to Kafka
  • DynamoDB Streams: captures item-level changes (OLD_IMAGE, NEW_IMAGE, or BOTH)
  • Lambda triggers or Kinesis Data Streams consume the stream
  • Retention: 24 hours
  • Spanner Change Streams: monitor changes to tables/columns
  • Consumed via Dataflow pipelines
    CREATE CHANGE STREAM MyStream FOR ALL;
    
  • Change Streams: resume token-based streaming from the oplog
    const changeStream = db.collection('orders').watch();
    changeStream.on('change', (change) => { console.log(change); });
    
  • Native CDC exposes selected commit-log segments; delivery to Kafka needs an additional integration
  • Evaluate connector ownership, version compatibility and support rather than assuming any Cassandra connector is a core Debezium connector

At-Least-Once Delivery

CDC systems typically guarantee at-least-once delivery. Consumers must be idempotent — processing the same event twice must have the same effect as processing it once.


Build a Recoverable Pipeline

A snapshot must hand off to the change stream without a gap. Persist offsets/resume tokens, preserve per-key ordering where required, handle schema evolution and retain enough source history to survive consumer outages. A transport stream alone is not an immutable audit archive: retention, permissions, sensitive fields and tamper resistance need separate design.

Use the transactional outbox when the goal is a business event: insert business state and an outbox row in the same transaction, then relay committed outbox rows. A raw row update says what changed, not necessarily the business intent. Consumers should atomically record deduplication state with their effects when possible.

Check Your Understanding

#

Question 1

Does exactly-once transport automatically make an external payment side effect exactly-once?

#

Question 2

How should you handle a replication slot retaining enough WAL to threaten disk capacity?