Publishing Outbox Events With Debezium: How CDC Reads Your Transaction Log and Keeps Kafka Events Ordered

The transactional outbox pattern commits a domain change and its event atomically, but a relay is still needed to move rows from the outbox table to the broker. An application-side poller is the common default, yet it adds query load, poll-interval latency, and publisher code that every service must own. Log-based change data capture removes the poller: Debezium tails the database's transaction log — the MySQL binlog or the PostgreSQL WAL — the way a replica client does, and forwards each committed outbox row to Kafka within milliseconds of the commit. This article examines that relay in detail: the replication mechanics it joins, the Single Message Transform that turns outbox rows into clean events, and why commit order combined with an aggregate identifier used as the Kafka message key guarantees per-entity ordering. It closes by comparing deployment paths — Kafka Connect, the embedded Debezium Engine, and managed CDC platforms — for teams that have already chosen an outbox and want a log-based publisher.

Why Log-Based Publishing Replaces the Application Poller

Benefits of a CDC‑based outbox relay

When the service writes an outbox row inside the same transaction that modifies the domain entity, the only component that changes is the relay that moves those rows to Kafka. A CDC tool such as Debezium attaches to the database as a logical replication client and reads the transaction log (WAL for PostgreSQL, binlog for MySQL) instead of issuing periodic SELECT statements against the outbox table.

• Near‑zero latency – Debezium picks up each committed change as soon as the log entry appears and forwards the event to Kafka within milliseconds of the commit, eliminating the deterministic wait introduced by a poll interval ​Source 1​.
• Zero additional read load – Because the connector consumes the log, the primary database is not repeatedly scanned for new rows; the workload is limited to the log streaming that already occurs for replication ​Source 2​.
• Exactly‑once change detection – The log guarantees that every committed row is seen once, so there is no need for “processed_at” flags, explicit locking, or duplicate‑row filtering that pollers usually require ​Source 3​.
• Resilience via offset tracking – Debezium records the last processed log position. If the connector crashes or restarts, it resumes from that offset, ensuring no outbox rows are missed ​Source 3​.
• Preserved ordering – The transaction log is inherently ordered by commit time; the connector emits events in that same order, which translates into per‑entity ordering when the outbox row’s identifier is used as the Kafka message key ​Source 4​.

Trade‑offs

Adopting a CDC‑based relay introduces operational responsibilities that a simple application poller does not have:

• Connector lifecycle management – Teams must deploy and monitor a Debezium connector (often via Kafka Connect) or embed the Debezium Engine.
• Enabling logical replication – The database must be configured to expose its transaction log to external clients (e.g., creating replication slots in PostgreSQL).

These steps add complexity, but the cost is usually justified when the system requires sub‑second event propagation or processes a high volume of outbox rows where polling would cause noticeable load or latency ​Source 5​.

When to choose CDC over a poller

• Event volume approaches or exceeds the throughput that periodic queries can sustain.
• Latency targets are tighter than typical poll intervals (e.g., sub‑100 ms).
• The organization already runs Kafka Connect or a managed CDC platform, making the extra component a natural fit.

In those scenarios, CDC removes the per‑service poller, its bookkeeping tables, and associated locking logic, while delivering durable events almost instantly after the originating transaction commits.

How Debezium Reads the Transaction Log Like a Replica Client

How Debezium Reads the Transaction Log Like a Replica Client

Debezium’s CDC connector does not poll the outbox table. Instead it registers as a replica client and consumes the same stream that the database uses to keep its backup replicas up‑to‑date.

• MySQL – The connector opens a binlog client session and follows the MySQL binary log using the ROW‑based format. This is the same protocol that MySQL replication slaves use.
• PostgreSQL – Debezium creates a logical replication slot and uses the pgoutput decoder (or another logical‑decoding plug‑in) to read the WAL as a logical stream. The slot guarantees that every committed change is retained until Debezium acknowledges it.

Because the log is already being written by the primary server, Debzie­um adds virtually no extra load: it never issues SELECT statements against the outbox table, it simply decodes each log entry that describes an INSERT, UPDATE, or DELETE.

Startup sequence

1. Snapshot – On first start Debezium takes a snapshot of the configured tables (usually just the outbox) to capture the initial state. The snapshot is written as a batch of change events and establishes the baseline for the stream.

2. Switch to streaming – Immediately after the snapshot the connector switches to the logical‑decoding stream. From this point on every committed transaction appears as a change event that includes:

   • transaction identifier,
   • log position (LSN for PostgreSQL, binlog file + position for MySQL),
   • commit timestamp, and
   • the before/after column values.

   The connector then applies any configured Single Message Transform to turn the raw row into a clean outbox event before writing it to Kafka.

Offset tracking

Debezium persists the last processed log position in its offset storage (Kafka Connect’s internal OFFSET_STORAGE_TOPIC or an external file when using the embedded engine). On restart it reads this offset and resumes from the next LSN or binlog coordinate, ensuring no duplicate or missing events even across fail‑overs.

Why the outbox stays narrow

The outbox is an append‑only table that receives only new rows. When the connector is limited to that table, the change set it has to decode is small, which reduces CPU and network usage compared with capturing the whole database.

Version‑sensitive details

Snapshot modes (initial, never, schema_only), the choice of logical decoder (pgoutput vs. wal2json), and default connector properties differ between Debezium releases and between MySQL 5.7/8.0 or PostgreSQL 12/13+. Readers should verify the exact defaults for their connector version in the official Debezium documentation.

“Debezium reads the transaction log of each supported database… The CDC connector acts like a replica client, connecting using the same protocol that logical replication uses” – Debezium blog, 2026【8†source】
“Debezium takes a snapshot … then reads the PostgreSQL logical decoding stream … each change is emitted as a change event record” – Transactional Outbox Pattern with Debezium【1†source】
“The connector stores the last processed log position … so it can resume safely after restarts” – Change Data Capture (CDC) in Modern Systems【2†source】

Database Prerequisites: Binlog Format, WAL Level, and Replication Slots

MySQL – Binary Log Requirements

Setting	Recommended Value	Why it matters
log_bin	ON	Enables the binary log so Debezium can read changes.
binlog_format	ROW	Row‑level events contain the complete after‑image of each changed row, which the connector translates into outbox events.
binlog_expire_logs_seconds	≥ 86400 (24 h) or larger for your SLA	Controls how long the binlog is kept; enough retention prevents the connector from falling behind after a short outage.
max_binlog_size	default (1 GB) is usually fine, but monitor the size of each file if you expect bursts of outbox rows.	Prevents excessive disk usage per file; Debezium reads sequentially, so file boundaries are not a bottleneck.

Privileges – The Debezium user must have at least:

GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'debezium'@'%' IDENTIFIED BY '••••••';

Do not grant SUPER unless absolutely required; a dedicated least‑privilege account reduces risk.

Operational tip – Enable a monitoring alert on Seconds_Behind_Master (or its equivalent in MySQL 8.0+) and on the binlog disk usage. This surfaces slot‑lag early, before the retention window is exhausted.

---

PostgreSQL – Logical Replication Prerequisites

Add the following to postgresql.conf (reload or restart afterwards):

wal_level = logical                 # activates logical decoding
max_replication_slots = 4          # one slot per Debezium connector (adjust to your scale)
max_wal_senders = 4                # parallel WAL sender processes
# Version‑sensitive caps (PostgreSQL 13+)
max_slot_wal_keep_size = 2GB      # optional: caps WAL retained for a slot
# Version‑sensitive timeout (PostgreSQL 18+)
idle_replication_slot_timeout = '5min'  # auto‑drop inactive slots

Create a minimal replication role:

CREATE ROLE debezium WITH LOGIN REPLICATION PASSWORD '••••••';
GRANT SELECT ON ALL TABLES IN SCHEMA public TO debezium;

Publication for the outbox table only (reduces noise and prevents unrelated schema changes from being captured):

CREATE PUBLICATION outbox_pub
    FOR TABLE public.outbox
    WITH (publish = 'insert, update, delete');

If you prefer a filtered approach, set publication.autocreate.mode = filtered (default in recent PostgreSQL releases) and list the table explicitly via table.include.list=public.outbox in the connector configuration JSON.

Replication slot – Debezium will create (or reuse) a logical slot, e.g. orders_outbox_slot. The slot must exist and be owned by the debezium role, or the connector can be configured to auto‑create it with the slot.name property.

---

Common Recommendations

• Dedicated user – Use a separate low‑privilege user for each connector rather than a superuser; this limits blast radius if credentials are compromised.
• Retention & alerting from day 1 – Set binlog_expire_logs_seconds (MySQL) or max_slot_wal_keep_size (PostgreSQL 13+) and configure alerts on WAL growth and slot lag.
• Scope the connector – In the Debezium JSON, include "table.include.list":"public.outbox" (PostgreSQL) or "table.include.list":"outbox" (MySQL) so only outbox rows are streamed. This avoids unnecessary load and keeps ordering guarantees tight.

By satisfying these engine‑specific prerequisites, the CDC pipeline can reliably read the transaction log, preserve commit order, and emit outbox events to Kafka with millisecond latency.

The Outbox Event Router: Turning Rows Into Kafka Records

The Outbox Event Router: Turning Rows Into Kafka Records

Debezium emits each change as a raw CDC envelope that contains before, after, source, and op fields. Without further processing, consumers would have to unwrap this structure themselves. The Outbox Event Router Single Message Transform (SMT) – enabled with

transforms.outbox.type=io.debezium.transforms.outbox.EventRouter

does the unwrapping automatically. It reads the configured columns of the outbox table, discards the envelope, and builds a clean Kafka record whose value is the plain JSON payload, key is the aggregate identifier, and whose type column is added as a header for downstream routing 【1】.

Expected outbox schema and field remapping

The transform expects a table roughly shaped as:

Column	Example	Purpose
id (UUID)	c1f7…	Unique event identifier
aggregatetype	Book	Determines the logical stream
aggregateid	123e4567‑e89b‑12d3‑a456‑426614174000	Becomes the Kafka key
type	BOOK_CREATED	Copied to a Kafka header
payload (JSON/JSONB)	{ "title": "…", "author": "…" }	Becomes the Kafka value

All column names are customizable through the transforms.outbox.table.field.* properties; for example transforms.outbox.table.field.event.key=aggregate_id remaps the key column 【1】.

Topic routing

By default the SMT routes each event to a topic named

outbox.event.<aggregatetype>

so a row with aggregatetype = "Book" lands in outbox.event.Book 【6】. Alternative routing is possible:

• Route by another column – Set transforms.outbox.route.by.field=destination to use the value of a destination column as the routing key, yielding topics such as outbox.event.item 【8】.
• Force a single topic – transforms.outbox.route.topic.replacement=bookstore.events collapses all events into one topic, irrespective of aggregate type 【7】.

Handling deletions (tombstones)

When the application cleans up processed outbox rows, Debezium observes an INSERT followed by a DELETE. By default this would produce a tombstone record (a null value) that can increase traffic and trigger compaction. Setting

tombstones.on.delete=false

suppresses the follow‑up delete record, keeping the stream limited to the original event 【5】.

Version‑sensitive configuration

Debezium’s SMT properties and default topic naming have evolved across releases (e.g., new routing options in 2.5+). Therefore the exact property names and defaults should be verified against the EventRouter documentation for the connector version in use 【9】. This safeguards against mismatched configs when upgrading or deploying different Debezium versions.

Commit Order Plus the Kafka Key: How Per-Entity Ordering Works

Commit Order Plus the Kafka Key: How Per‑Entity Ordering Works

Transaction‑log ordering
Debezium consumes the database’s replication log (MySQL binlog or PostgreSQL WAL) exactly as the database writes it. Because the log records every transaction commit in sequence, Debezium streams outbox rows to Kafka in the same commit order, without any reordering introduced by the connector¹.

Kafka’s per‑key ordering guarantee
The EventRouter Single Message Transform (SMT) extracts the aggregate_id field from each outbox row and sets it as the Kafka record key. Kafka routes all records with the same key to a single partition and guarantees that they are delivered to consumers in the order they were produced to that partition²³. Consequently, all events for a particular aggregate—e.g., Order #123—appear on the same partition and are consumed in the exact order they were committed to the database.

Scope of the guarantee

• Inside a single aggregate: ordering is preserved as long as the key remains unchanged.
• Across aggregates: no ordering guarantee exists because different aggregate_ids are spread over multiple partitions to enable parallel consumption. This is intentional; it allows the consumer group to scale horizontally.

Failure modes that break ordering

Issue	Impact on ordering
Changing the partition key after some events have been produced	Subsequent events may be routed to a different partition, splitting the aggregate’s stream.
Re‑using identifiers across aggregate types (e.g., a plain numeric ID for both Order and Invoice)	Colliding keys cause events for unrelated entities to share a partition, interleaving their order.
Mutating aggregate_id after the first event	The same logical entity ends up on multiple partitions, destroying the per‑entity sequence.

Safeguards

• Keep aggregate_id immutable and globally unique—using a UUID (or a UUID prefixed with the aggregate type) eliminates collisions.
• If downstream consumers need an extra deterministic sequence, add a monotonically increasing aggregate_version column and enforce a unique index on (aggregate_type, aggregate_id, aggregate_version).
• Configure the Debezium SMT to always map aggregate_id to the record key (transforms.outbox.table.field.event.key=aggregate_id).

Consumer‑side best practice
Disable Kafka’s enable.auto.commit. Instead, process each record and commit the consumer offset only after successful handling. This ensures that on a restart the consumer does not skip an unprocessed event, preserving the commit‑order guarantee on the consumer side.

At-Least-Once Delivery and Idempotent Consumers

At-Least-Once Delivery and Idempotent Consumers

Debezium guarantees at‑least‑once delivery. It reads every change that has been committed to the database’s transaction log and forwards the record to Kafka. If the connector crashes, it restarts from the last stored log offset, ensuring that no committed change is lost — but a record that was sent to Kafka before the offset was flushed can be emitted again, so duplicates are always possible【{"id":0,"cursor":0}】.

Because the pipeline does not provide exactly‑once guarantees, downstream services must be idempotent. Processing the same OrderCreated event twice must never cause a second shipment, a duplicate invoice, or any other unintended side‑effect. Common techniques are:

• Duplicate‑key check – before handling the event, query a table (or cache) to see whether the aggregate identifier has already been processed.
• Deduplication store – persist a unique event ID (e.g., the outbox row’s primary key or a UUID) in a durable store; skip processing if the ID exists【{"id":0,"cursor":1}】.
• Naturally idempotent operations – design the business logic so that re‑applying the same command has no additional effect (e.g., “create shipping record if not exists”).

These patterns eliminate the risk of double side‑effects even when Debezium re‑emits a record.

Offset mismanagement as a failure mode

Offsets are stored by Kafka Connect in a dedicated offset storage topic (OFFSET_STORAGE_TOPIC). If this topic is corrupted or deleted, the connector falls back to the earliest available position and may replay a large volume of already‑processed events, overwhelming downstream consumers. Conversely, writing an offset before the downstream system has successfully processed the corresponding event can mask an undelivered message, because the connector will consider the record handled and will not retry it【{"id":2,"cursor":2}】.

Practical recommendations

• Durable offset storage – keep offsets in Kafka Connect’s internal offset topic rather than local files; this topic is replicated and participates in Kafka’s transactional guarantees【{"id":2,"cursor":3}】.
• Offset monitoring – track offset lag and set alerts for sudden jumps or stalls, which often indicate connector restarts or storage issues.
• Never assume exactly‑once – the contract of the outbox + Debezium pipeline is per‑entity ordering and idempotent consumers; design your services accordingly.

By treating every incoming record as potentially duplicated and by persisting processing state atomically with business updates, teams can reliably build event‑driven systems on top of Debezium’s at‑least‑once semantics.

Operational Hazards: WAL Retention, Slot Lag, and Table Bloat

Operational Hazards: WAL Retention, Slot Lag, and Table Bloat

Replication‑slot‑driven WAL growth (PostgreSQL)
When a Debezium connector consumes the outbox via logical decoding it creates a replication slot. The slot prevents PostgreSQL from recycling any WAL segment that it has not yet acknowledged. If the connector falls behind or stops, pg_wal retains those segments, quickly filling the primary’s disk. Once the disk is exhausted PostgreSQL refuses new writes, turning a lagging CDC consumer into a system‑wide outage [1][2].

Mitigations

Setting	Purpose	Example
max_slot_wal_keep_size (Postgres 13+)	Caps the total WAL size a slot may hold; generates alerts when the limit is reached	max_slot_wal_keep_size = 2GB
idle_replication_slot_timeout (Postgres 18+)	Drops slots that have been inactive for the configured period, preventing forgotten slots from consuming WAL	idle_replication_slot_timeout = '5min'
Monitoring pg_replication_slots.restart_lsn vs. pg_current_wal_lsn	Shows how far the consumer is behind; alert when the lag exceeds a threshold	SELECT slot_name, pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) AS lag_bytes FROM pg_replication_slots;

MySQL binlog retention
The MySQL connector behaves similarly: it can only resume from binlog files that still exist. If the server rotates or expires binlogs before the CDC consumer catches up, the connector loses its position and must start with a fresh snapshot, risking data loss [6]. Configure binlog_expire_logs_seconds (or the equivalent) to keep logs longer than any expected downtime and monitor the connector’s LSN lag.

Outbox table bloat
Every DELETE (or UPDATE that marks a row as published) creates a dead tuple that autovacuum must reclaim. High delete rates can generate hundreds of thousands of dead tuples per minute, outpacing the default autovacuum_vacuum_cost_limit. The result is rapid table growth, index bloat, and slower scans [9].

A low xmin horizon—caused by a long‑running transaction, a hung replica, or hot‑standby feedback—prevents vacuum from cleaning any dead rows, allowing the outbox to balloon from megabytes to dozens of gigabytes [10].

Poison‑pill rows
A malformed outbox entry (e.g., invalid JSON or missing required field) can cause the connector to retry indefinitely. Because CDC processes rows sequentially, a single “poison pill” blocks subsequent rows, turning the outbox into an unbounded queue [11].

Operational recommendations

• Continuous lag monitoring: track replication‑slot WAL lag (restart_lsn vs. pg_current_wal_lsn), MySQL binlog position, and connector health metrics from day 1.
• Alert on growth thresholds: disk usage of pg_wal, dead‑tuple count (pg_stat_all_tables.n_dead_tup), and outbox depth.
• Capacity‑tuned autovacuum: raise autovacuum_vacuum_cost_limit, lower autovacuum_vacuum_scale_factor, and ensure no long‑running transactions keep xmin low.
• Retention safeguards: set max_slot_wal_keep_size (Postgres 13+), idle_replication_slot_timeout (Postgres 18+), and a generous binlog_expire_logs_seconds for MySQL.
• Recovery playbook: include steps to drop and recreate stale replication slots, resnapshot the connector, and purge or reprocess poison‑pill rows.

Treat the CDC pipeline as production‑critical infrastructure: its failure not only delays event publishing but also back‑pressures the primary database, potentially causing the very outages the transactional outbox pattern is meant to avoid.

Deployment Models: Kafka Connect, the Embedded Engine, or Managed CDC

Deployment Models: Kafka Connect, the Embedded Engine, or Managed CDC

Kafka Connect is the default, production‑grade way to run Debezium. In a Connect cluster the Debezium source connector runs as a worker task that automatically rebalances, persists offsets, and recovers from failures. Because Connect ships with a huge catalog of sink connectors (JDBC, Elasticsearch, S3, Snowflake, BigQuery, etc.), a single configuration can fan‑out outbox events to many downstream systems without custom code【{"cursor":0,"id":2}】. This model works best when a Kafka ecosystem already exists and you need the built‑in high‑availability and scaling guarantees that distributed Connect provides.

The Embedded Debezium Engine is a low‑level Java API that starts a connector programmatically from any application. It gives you full control over threading, lifecycle, and error handling, and is the foundation for framework integrations such as Quarkus, Spring and Camel【{"cursor":0,"id":3}】. Use the Engine when you want CDC to live inside a single microservice or a bespoke data‑pipeline and you prefer not to introduce a separate Connect cluster. The trade‑off is that offset storage, fault tolerance and scaling become the responsibility of the host application, so you must implement durable offset handling and restart logic yourself.

Debezium Server (and the newer Debezium Platform operator) offers a container‑ready, stand‑alone runtime that streams the same source connectors to non‑Kafka destinations such as Amazon Kinesis, Google Cloud Pub/Sub, Apache Pulsar, Redis Streams, RabbitMQ or HTTP webhooks【{"cursor":0,"id":0}】. This eliminates the need for any Kafka brokers while preserving the connector logic. It is suited for teams that already use these clouds or messaging systems and want a minimal footprint.

For organizations that do not want to manage any CDC infrastructure, managed services such as Streamkap provide a fully hosted Debezium pipeline. Streamkap connects directly to the source database, creates and monitors replication slots, streams changes to Kafka, and automatically handles schema evolution, lag monitoring and auto‑restarts【{"cursor":1,"id":0}】. The cost is vendor lock‑in and reduced low‑level control.

Decision Guidance

Situation	Recommended model
Existing Kafka cluster, need robust HA and many sink integrations	Kafka Connect
CDC must be embedded in a single service, no extra infrastructure	Embedded Engine (manage offsets yourself)
Want CDC without Kafka, targeting other cloud streams	Debezium Server / Platform
No team owns CDC ops, prefer a turn‑key solution	Managed CDC (e.g., Streamkap)

Note: The feature set of each deployment option evolves across Debezium releases; always verify the current release notes before committing to a model.

Methodology and sources

• Reviewed: October 22, 2026.
• Method: Documentation-led analysis using the primary sources listed below. Validate versions and environment-specific claims before production use.
• Debezium MySQL Connector Tutorial — Official Documentation
• What Nobody Explains About Debezium — Deployment Models and Prerequisites
• Outbox Pattern with CDC and Debezium — Thorben Janssen
• Outbox Pattern and CDC for Reliable Event Publishing in Microservices — Streamkap
• Outbox Pattern for Reliable Event Publishing — Conduktor Glossary
• Transactional Outbox Pattern with Debezium, Kafka, and PostgreSQL — Florian Courouge
• The Transactional Outbox Is Not a Queue — WAL Retention and Slot Hazards

Footnotes

1. Rabinarayan Patra, “Implementing Outbox Pattern with CDC in Microservices,” Rabinarayan Patra Blog, 2025. https://www.rabinarayanpatra.com/blogs/implementing-outbox-pattern-cdc-microservices ↩

2. Florian Courouge, “Transactional Outbox Pattern with Debezium, Kafka, and PostgreSQL,” Florian Courouge Blog, 2024. https://floriancourouge.com/en/blog/transactional-outbox-pattern-debezium-kafka-postgres ↩

3. StreamKap, “Outbox Pattern and CDC for Reliable Event Publishing in Microservices,” StreamKap Resources, 2025. https://streamkap.com/resources-and-guides/outbox-pattern-explained ↩