18 · Change Data Capture (CDC) — Stream Database Changes Downstream¶
Distributed Transactions · Topic 18 of 20
What is CDC?¶
Change Data Capture tracks row-level changes (INSERT, UPDATE, DELETE) in a database and streams them to downstream consumers in real time.
CDC turns the database into an event stream.
Use Cases¶
| Use Case | How CDC Helps |
|---|---|
| Search index sync | Sync Elasticsearch on every DB write |
| Cache invalidation | Invalidate Redis entries when data changes |
| Audit logging | Immutable log of all data changes |
| Data replication | Replicate to another DB or data warehouse |
| Event-driven pipelines | Trigger microservices on data changes |
How CDC Works¶
sequenceDiagram
App->>Database: INSERT / UPDATE / DELETE
Database->>WAL: Append change record
CDC Tool->>WAL: Read changes (tail the log)
CDC Tool->>Kafka: Publish change events
Consumers->>Kafka: Read and process events
Log-based CDC, illustrated above, reads the WAL or equivalent committed-change stream. CDC is a broader category that also includes trigger-based capture and polling; those approaches have different overhead and correctness trade-offs.
CDC Tools¶
| Tool | Source DBs | Target |
|---|---|---|
| Debezium | PostgreSQL, MySQL, MongoDB, SQL Server | Kafka, Kinesis |
| AWS DMS | Most relational DBs | S3, Kinesis, RDS |
| Striim | Multi-source | Kafka, GCS, BigQuery |
| Datastream (GCP) | PostgreSQL, MySQL, Oracle | BigQuery, GCS, Spanner |
Cloud Implementations¶
- Enable logical replication:
wal_level = logical - Create replication slot:
pg_create_logical_replication_slot('debezium', 'pgoutput') - Debezium reads from the slot and publishes to Kafka
- DynamoDB Streams: captures item-level changes (OLD_IMAGE, NEW_IMAGE, or BOTH)
- Lambda triggers or Kinesis Data Streams consume the stream
- Retention: 24 hours
- Spanner Change Streams: monitor changes to tables/columns
- Consumed via Dataflow pipelines
- Change Streams: resume token-based streaming from the oplog
- Native CDC exposes selected commit-log segments; delivery to Kafka needs an additional integration
- Evaluate connector ownership, version compatibility and support rather than assuming any Cassandra connector is a core Debezium connector
At-Least-Once Delivery¶
CDC systems typically guarantee at-least-once delivery. Consumers must be idempotent — processing the same event twice must have the same effect as processing it once.
Build a Recoverable Pipeline¶
A snapshot must hand off to the change stream without a gap. Persist offsets/resume tokens, preserve per-key ordering where required, handle schema evolution and retain enough source history to survive consumer outages. A transport stream alone is not an immutable audit archive: retention, permissions, sensitive fields and tamper resistance need separate design.
Use the transactional outbox when the goal is a business event: insert business state and an outbox row in the same transaction, then relay committed outbox rows. A raw row update says what changed, not necessarily the business intent. Consumers should atomically record deduplication state with their effects when possible.
Check Your Understanding¶
Question 1
Does exactly-once transport automatically make an external payment side effect exactly-once?
Question 2
How should you handle a replication slot retaining enough WAL to threaten disk capacity?