What is Change Data Capture used for?

Change Data Capture is used to turn database changes into a continuous stream of events. Instead of polling whole tables, it captures committed inserts, updates and deletes and sends them to systems such as Kafka. Common uses include refreshing analytics tables, syncing caches, updating search indexes and feeding event-driven services.

Why read the transaction log for CDC?

The transaction log records the committed changes the database already depends on for recovery and replication. Reading it usually adds less application overhead than triggers and is more reliable than timestamp polling. It also gives a clearer view of inserts, updates and deletes in the order the database committed them.

What can go wrong in a CDC pipeline?

CDC pipelines can produce duplicate events, lose ordering guarantees across partitions, mishandle deletes, or break when source schemas change. They also need an initial snapshot so downstream state starts complete. Production designs should track connector lag and failures, and make consumers idempotent using keys plus version or timestamp fields.

Streaming & Real-Time

05.06 · Walkthrough

Change Data Capture

Use a CDC pattern to capture database inserts, updates, and deletes and publish them into a streaming pipeline.

Change Data Capture publishes committed row changes from a database into a stream, so downstream systems can react to inserts, updates and deletes without full-table scans. A solid CDC design handles transaction-log reading, initial snapshots, duplicate delivery, ordering, deletes, schema evolution and connector failure modes.

What this lesson answers

how does change data capture work
cdc transaction log versus database triggers
how to handle deletes in cdc pipelines

Notes

Change Data Capture, or CDC, is a pattern for detecting row-level changes in a source database and publishing those changes into another system, often a stream. Instead of repeatedly scanning full tables or relying on batch exports, CDC captures inserts, updates, and deletes as they happen or shortly after they are committed. In a real-time data architecture, this lets downstream systems react quickly to operational changes: updating search indexes, refreshing analytics tables, syncing caches, feeding event-driven services, or maintaining data lake replicas.

A common CDC implementation reads…

Common questions

What is Change Data Capture used for?: Change Data Capture is used to turn database changes into a continuous stream of events. Instead of polling whole tables, it captures committed inserts, updates and deletes and sends them to systems such as Kafka. Common uses include refreshing analytics tables, syncing caches, updating search indexes and feeding event-driven services.
Why read the transaction log for CDC?: The transaction log records the committed changes the database already depends on for recovery and replication. Reading it usually adds less application overhead than triggers and is more reliable than timestamp polling. It also gives a clearer view of inserts, updates and deletes in the order the database committed them.
What can go wrong in a CDC pipeline?: CDC pipelines can produce duplicate events, lose ordering guarantees across partitions, mishandle deletes, or break when source schemas change. They also need an initial snapshot so downstream state starts complete. Production designs should track connector lag and failures, and make consumers idempotent using keys plus version or timestamp fields.

Short definition: what is Change Data Capture?

05.05 · Streaming & Real-TimeStreaming Features for ML 05.07 · Streaming & Real-TimeBackpressure, Replay, and Failure Recovery