Change Data Capture

Change Data Capture (CDC) is the process of tracking and capturing row-level changes (inserts, updates, deletes) as they occur in a source database, and streaming those changes to downstream systems in near real time. In ecommerce, CDC is the technical foundation for real-time inventory sync, event-driven order processing, Strangler Pattern migrations, and polyglot data architectures where the same business event must feed multiple specialised stores (search indexes, analytics, caches, CDPs).

---

Three CDC methods

Confluent's official learning documentation (© 2014–2026) describes three approaches:

Method	How it works	Pros	Cons
Timestamp-based	Polls a LAST_MODIFIED column on a schedule	Simple to implement; no tooling required	Cannot capture hard DELETEs; adds query load; requires schema change (adding the column)
Trigger-based	DB triggers fire on DML and write to a shadow/event table	Captures all change types including deletes	Impacts source DB write performance; requires schema change; can become complex at scale
Log-based	Reads the database transaction log (binlog for MySQL, WAL for PostgreSQL)	No overhead on source; captures all change types; no schema change needed	Log format is database-vendor-specific and can change between releases; consumers must filter rolled-back transactions

Log-based CDC has become the standard for high-throughput ecommerce use cases. Confluent describes it as "the de facto method for migrating to the cloud" (© 2014–2026).

---

How log-based CDC works (Debezium)

Debezium (Red Hat-backed open-source project, stable v3.6 as-of 2026) is the dominant open-source CDC framework. Its official documentation describes the architecture:

• Debezium runs as a set of source connectors for Apache Kafka Connect. Each connector reads from one source database: MySQL reads from the binlog; PostgreSQL reads from a logical replication stream.
• On startup, the connector takes an initial snapshot of existing data, then switches to streaming mode to capture ongoing changes.
• Each change event carries the before and after state of the row, the operation type (c = insert, u = update, d = delete, r = read/snapshot), the table name, and a precise timestamp.
• Events are published to Kafka topics named by convention: <prefix>.<schema>.<table> (e.g. appdb.public.orders). Topic routing SMTs (Single Message Transforms) allow redirection.
• Three deployment modes: (a) Kafka Connect (primary); (b) Debezium Server (no Kafka required — emits to Amazon Kinesis, Google Cloud Pub/Sub, Apache Pulsar); (c) Debezium Engine (embedded in a JVM application).

Debezium supports the following source databases (nightly build, as-of 2026): MySQL, MariaDB, MongoDB, PostgreSQL, Oracle, SQL Server, Db2, Cassandra, Vitess, Spanner, Informix, CockroachDB, YashanDB, Ingres. Sink connectors: JDBC, MongoDB Sink.

Debezium guarantees at-least-once delivery — not exactly-once. Consumers must implement deduplication or idempotent processing. (Debezium official docs, stable v3.6, as-of 2026.)

---

Ecommerce-specific use cases

Conduktor (Stéphane Derosiaux, July 30, 2026) and TopETL (Ava Mercer, March 4, 2026) identify six primary CDC pipeline use cases in ecommerce:

1. Real-time order feed — CDC streams state transitions (PENDING → CONFIRMED → SHIPPED → DELIVERED) as before/after events; downstream services (inventory, notifications, payments, analytics) consume independently from Kafka without requiring a shared ACID transaction across services.
2. Inventory sync — when a product is sold, CDC captures the inventory row update and propagates it to the storefront, preventing overselling and backorders. (Macrometa, © 2026.)
3. Marketing spend / ROAS unification — near real-time revenue feed from the order DB joined against ad spend events, enabling same-session ROAS measurement rather than T+1 batch reporting.
4. Strangler Pattern migrations — CDC streams changes from a legacy database to Kafka during the transition period; new microservices consume the same stream without requiring application changes to the monolith. See Strangler Pattern for the full pattern.
5. CQRS and polyglot persistence — CDC streams product catalog changes from PostgreSQL to Elasticsearch for search, to Redis for cache, and to a data warehouse for analytics. Each read model is optimised for its access pattern without hitting the transactional OMS database. (Conduktor, July 30, 2026.)
6. Identity Resolution and CDP streams — near real-time customer profile updates fed from the order and account DBs into the CDP or ESP, enabling lifecycle messaging triggered by actual behaviour rather than batched segment refreshes.

---

The outbox pattern

CDC solves the dual-write problem inherent in event-driven microservices: writing to a database and publishing a Kafka event are two separate operations, and a crash between them creates inconsistency. The outbox pattern (Conduktor, July 30, 2026):

1. The application writes both the business data and a row to an outbox table atomically in a single database transaction.
2. A CDC connector (Debezium) monitors the outbox table and streams each row to Kafka.
3. Downstream services consume from Kafka; the outbox row is marked processed.

This preserves causal ordering from the database transaction log and achieves millisecond latency from change to event availability in Kafka. Debezium ships a built-in Outbox Event Router SMT for this pattern.

---

CDC in platform migrations (ecommerce context)

The Strangler Pattern article documents CDC as the most underestimated step in monolith decomposition. Conduktor (July 30, 2026) describes the three-phase CDC-backed migration:

• Phase 1 — Event infrastructure: Deploy Debezium to stream legacy DB changes into Kafka. The legacy system continues unchanged; new services are built alongside it consuming from Kafka.
• Phase 2 — Shadow mode: New services consume the same Kafka events but run in parallel without serving traffic; their output is compared against the legacy system.
• Phase 3 — Cutover: Traffic is switched to new services; CDC continues until the legacy DB is decommissioned.

Schema management is the hardest problem. A schema registry (Apicurio, AWS Glue Schema Registry, or Confluent Schema Registry) is required to enforce compatibility rules. Recommended approach for column renames during a migration: add the new column alongside the old, run CDC capturing both, allow consumers to migrate at their own pace, then deprecate the old column. (Conduktor, July 30, 2026.)

martinfowler.com (Cartwright, Horn, Lewis — Thoughtworks, March 2024) notes: "CDC encompasses technologies that allow you to create an event stream from entries appended to a database's transaction log. For example our teams have had good experiences using Debezium to create a Kafka event stream that can be consumed by new applications." CDC is positioned as a key enabler of incremental domain extraction in legacy displacement programmes.

---

SaaS platforms: the Subscriptions API as CDC equivalent

Not all ecommerce databases are accessible for log-based CDC. SaaS platforms like commercetools do not expose raw database access. Their architectural CDC equivalent is the Subscriptions API (commercetools official docs, © 2026):

• Subscriptions notify external systems of real-time events (e.g. Order placed, Cart updated).
• Supported delivery destinations: AWS SQS, Google Cloud Pub/Sub, Azure Service Bus.
• Delivery guarantee: at-least-once — consumers must be idempotent.
• Message ordering: not guaranteed chronologically across the system — use the sequenceNumber field to process in-order per resource.
• Message size cap: if payload exceeds messaging system size limits (e.g. 256 KB for SQS), the platform truncates — preserving metadata but removing the resource body. Consumers must detect truncation and fetch the full resource via GET.
• Max 50 Subscriptions per project; use SNS fan-out for multiple consumers.
• CloudEvents 1.0 specification supported (since December 2019).
• Checkout Events (public beta, June 2025): event notifications during Order creation and Payment processing.

---

At scale: Shopify's CDC implementation

The most detailed ecommerce CDC case study in public literature is Shopify's 2021 migration from a query-based tool ("Longboat") to log-based CDC using Debezium across their sharded MySQL monolith. (Shopify Engineering blog, March 12, 2021 — architecture may have evolved; included as canonical practitioner reference.)

Why Longboat failed:

• Max data freshness: 1 hour due to query performance constraints
• Could not capture hard deletes (no updated_at to detect removal)
• Missed intermediate row states between job runs
• Consistency issues when joining snapshots across shards

What Debezium replaced it with:

• One Debezium connector per MySQL shard (~100+ shards)
• RegexRouter SMT merges all shard events → per-table compacted topics partitioned by primary key
• Compacted topics allow consumers to initialise local state from the latest record per primary key (full snapshot via Kafka)
• Confluent Schema Registry as schema catalog and data discovery layer

Production metrics (as-of Black Friday 2020):

• p99 latency from MySQL insertion to Kafka availability: <10 seconds (median much lower; affected by MySQL read-replica replication lag)
• Peak throughput: 100,000 records/second; sustained: 65,000 records/second
• CDC cluster size: 400 TB+ of CDC data; ~150 Debezium connectors on 12 Kubernetes pods

Impact: Marketing Engagements API moved from batch warehouse model to CDC stream → average data freshness improved from 1 day to 1 hour. (Shopify Engineering, March 12, 2021.)

---

Benchmarks

Metric	Value	Source
Debezium single-threaded throughput (PostgreSQL connector)	~7,000 events/second	Estuary.dev (Dani Pálma, Apr 2025/Apr 2026) — vendor-authored; test version unspecified
Shopify peak throughput (MySQL, 100+ shards)	100,000 records/second	Shopify Engineering (Mar 2021) — historical
Shopify p99 MySQL-to-Kafka latency	<10 seconds	Shopify Engineering (Mar 2021) — affected by read-replica lag
Slack batch-to-CDC latency improvement	24 hours → <10 minutes	Confluent Current 2024 (Oct 2024) — Slack practitioner case study
Glossier sync improvement (Shopify ERP)	4 hours → minutes	Estuary.dev success story (© 2026) — vendor case study

---

Challenges and operational complexity

Estuary.dev (Dani Pálma, Apr 2025/Apr 2026 — note: vendor with commercial interest) identifies the following Debezium production challenges:

• Single-threaded initial snapshot: non-concurrent; may lock tables on large datasets
• At-least-once delivery only: consumers must deduplicate
• Schema changes: renaming columns or altering primary keys requires manual connector restarts and potentially a full re-snapshot
• PostgreSQL WAL bloat: if a replication slot falls behind, unread WAL segments accumulate and can exhaust disk — a severe operational risk
• Large records: Shopify worked around this by compressing records >1 MB and storing very large blobs in GCS with a pointer record in Kafka

---

Observability

Confluent/Conduktor (July 30, 2026) recommend monitoring these key CDC metrics in production:

• CDC lag: time between database change and Kafka event availability
• Replication slot growth (PostgreSQL): WAL segments retained for CDC — disk exhaustion risk if Debezium connector lags
• Consumer group lag: use Kafka Lag Exporter or Burrow
• Schema compatibility alerts: from the schema registry
• Connector throughput and error rates

Recommended observability stack (as-of 2026): OpenTelemetry + Prometheus + Grafana.

---

Key tools (as-of March 2026)

Nine major CDC-enabled pipeline options identified by TopETL (Ava Mercer, March 4, 2026):

Tool	Type	Notes
Debezium	Open-source (DIY)	Kafka-native; full control; requires team to operate; log-based
Fivetran	Managed SaaS	Common in ecommerce data warehouse pipelines
Airbyte	Open-source + managed	Popular for warehouse-centric CDC
Google Cloud Datastream	Serverless managed	Oracle, MySQL, PostgreSQL → BigQuery; GCP-centric retailers
AWS DMS	Managed cloud	Broad DB support; MS-Replication + MS-CDC for SQL Server; logical replication for PostgreSQL
Striim	Commercial	Real-time streaming + transformation
Qlik Replicate	Commercial	Enterprise data replication
Integrate.io	SaaS	ETL/ELT with CDC modes
Hevo Data	SaaS	Automated pipelines

---

Key terms

Term	Meaning
WAL (Write-Ahead Log)	PostgreSQL's transaction log, read by Debezium for CDC
Binlog	MySQL's binary log equivalent, read by Debezium for CDC
Replication slot	PostgreSQL mechanism that retains WAL until a consumer has read it — can cause disk exhaustion if the consumer lags
Outbox pattern	Writing business data + event to an outbox table atomically; CDC then streams the event — avoids dual-write inconsistency
CQRS	Command Query Responsibility Segregation — CDC enables streaming from write model to specialised read models
Compacted topic	Kafka topic that retains only the latest record per key — enables consumer initialisation from Kafka without needing the full database
Debezium Server	Kafka-free Debezium deployment that emits directly to Kinesis, Pub/Sub, or Pulsar
Subscriptions API	SaaS platform equivalent of CDC (e.g. commercetools) — at-least-once event delivery to external message queues
CDC lag	Time delta between a database change occurring and the corresponding Kafka event being available to consumers
Shadow mode	Running a new service in parallel with a legacy system, consuming CDC events, to validate before traffic cutover

---

Next frontier

• Kafka (the messaging backbone for log-based CDC; referenced throughout but no dedicated concept page)
• Event-Driven Architecture (parent pattern; no dedicated concept page)
• Outbox Pattern (referenced here and in Strangler Pattern; no dedicated concept page)
• CQRS (Command Query Responsibility Segregation; referenced here; no dedicated concept page)
• Domain-Driven Design (DDD) (bounded context seam identification is a prerequisite for deciding which tables to CDC; 4+ vault refs; no page)
• Digital Experience Platform (DXP) (Gartner 2025 MQ; composability mandatory; 14 total refs; no page)