The Transactional Outbox Pattern: Solving the Dual-Write Problem Before It Corrupts Your Microservices Data

I have seen more microservices systems drop events silently than I care to count. The service looks healthy. The logs show success. The database row got updated. But somewhere downstream, the event that was supposed to trigger the fulfillment workflow, the email notification, or the inventory update never arrived. The dual-write problem is quiet, intermittent, and capable of corrupting your data in ways that take weeks to notice.

After twenty years of building distributed systems, I can tell you that the transactional outbox pattern is one of the few genuinely simple solutions to a genuinely hard problem. It does not require distributed transactions. It does not require saga orchestration. It does not require exotic databases. It requires discipline, a single extra table, and the right relay mechanism, and in exchange it gives you reliable event publishing with the same atomicity guarantee your database already provides.

The Dual-Write Problem in Plain Language

Consider a straightforward order placement flow. An HTTP request arrives. Your service writes a new order record to PostgreSQL. Then it publishes an OrderPlaced event to Kafka so the fulfillment service can pick it up. Two writes. Two systems. Zero atomicity between them.

Here is what can go wrong:

1. The PostgreSQL write succeeds, the Kafka publish fails. The order exists in your database but the event is lost. The fulfillment service never wakes up. The order sits in limbo until a human notices.
2. The Kafka publish succeeds, then the PostgreSQL write fails (maybe you hit a constraint). The event fires for an order that does not exist yet, or that gets rolled back. The fulfillment service processes a ghost order.
3. The service crashes, is killed by a deployment, or loses its network connection after the database write but before the publish call returns. Depending on the timing, the event may or may not have been sent.

The naive fix is to wrap the publish in a try-catch and retry. That helps with transient broker failures but does nothing for process crashes, and it introduces duplicates if the publish was actually delivered but the acknowledgement was lost. You can add idempotency keys to handle duplicates, but you still have not solved the fundamental race: the two writes are not atomic.

The other naive fix is to reverse the order. Publish first, then write to the database. This makes the ghost-order scenario the likely one instead of the lost-event scenario, which is usually worse.

You need atomicity across a database write and a message publish, and you do not want to reach for two-phase commit. The outbox pattern gives you that atomicity by reducing the problem: instead of two writes to two different systems, you make two writes to one system, and then let a relay handle the rest.

How the Outbox Pattern Works

The core idea is elegant. When your service processes a command, it does not write to both the database and the message broker. Instead, it writes everything to the database within a single transaction: the business entity update and a new row in a dedicated outbox table. The row describes the event that should be published, not the fact that it was published. Atomicity is guaranteed by your database. If the transaction commits, both the business entity change and the outbox row exist. If it rolls back, neither exists.

A separate relay process, running outside your service, is responsible for polling or streaming the outbox table and publishing those rows to the message broker. Once a row is successfully published and acknowledged, the relay marks it processed, or deletes it. The relay is a write-ahead log reader or a polling loop, not business logic. It does not know what the events mean. It just moves them.

This is not a new pattern. Chris Richardson documented it on microservices.io years ago. AWS has it in their Prescriptive Guidance for cloud design patterns. The reason I keep having to explain it is that teams think the dual-write problem is theoretical until it bites them, and then they often reach for the wrong fix.

The outbox table schema is simple. A minimal version in PostgreSQL looks like this:

CREATE TABLE outbox_events (
    id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    aggregatetype  TEXT NOT NULL,
    aggregateid    TEXT NOT NULL,
    type           TEXT NOT NULL,
    payload        JSONB NOT NULL,
    created_at     TIMESTAMPTZ NOT NULL DEFAULT now(),
    processed      BOOLEAN NOT NULL DEFAULT false
);

When you process an order placement, your transaction does exactly this:

BEGIN;

INSERT INTO orders (id, customer_id, status, total)
VALUES ($1, $2, 'placed', $3);

INSERT INTO outbox_events (aggregatetype, aggregateid, type, payload)
VALUES ('Order', $1, 'OrderPlaced', '{"orderId": ..., "customerId": ...}');

COMMIT;

Both inserts succeed or both roll back. Your relay picks up the outbox row and publishes the event to Kafka. The downstream service receives OrderPlaced and processes it. That is the entire pattern.

Now the question is: how should your relay work? There are two fundamentally different approaches, and the right choice depends on your scale, your operational maturity, and how much latency you can tolerate.

Approach 1: The Polling Publisher

The polling publisher is the simpler approach. You write a background thread or a separate service that runs on a timer, queries the outbox table for unprocessed rows, publishes each row to the broker, and marks the rows processed.

def poll_and_publish():
    while True:
        rows = db.query(
            "SELECT * FROM outbox_events WHERE processed = false ORDER BY created_at LIMIT 100 FOR UPDATE SKIP LOCKED"
        )
        for row in rows:
            producer.send(
                topic=f"{row.aggregatetype.lower()}.events",
                key=row.aggregateid,
                value=row.payload
            )
            producer.flush()
            db.execute(
                "UPDATE outbox_events SET processed = true WHERE id = %s",
                row.id
            )
        time.sleep(1)

The FOR UPDATE SKIP LOCKED clause is critical. It prevents two relay instances from processing the same row simultaneously when you run multiple replicas. Without it, you will publish duplicates at every horizontal scale event.

The polling approach has real advantages. It requires no additional infrastructure. It works on any relational database. It is easy to reason about, easy to debug, and easy to extend. If your outbox volume is manageable and your latency requirement is in the one-to-five second range, polling is entirely reasonable.

The drawbacks are latency and database load. You are running queries against your primary on a tight loop. Every active instance of your relay is hitting the database even if there is nothing to process. At high throughput, the SELECT and UPDATE queries on the outbox table add measurable IOPS to your primary, which matters when your primary is already under load from your application.

Polling is a good starting point for teams new to the pattern. You can get it working in an afternoon. You can switch to CDC later when your scale justifies the operational complexity.

Approach 2: CDC with Debezium

Change Data Capture (CDC) is a more sophisticated relay mechanism that reads directly from your database’s transaction log rather than polling the table. For PostgreSQL, this means reading from the Write-Ahead Log (WAL) via logical replication. I covered the fundamentals of WAL in our database write-ahead logging guide and the broader CDC landscape in our Debezium and CDC explainer, so I will focus here on how CDC changes the outbox relay specifically.

With CDC, Debezium acts as a PostgreSQL logical replication client. It subscribes to the WAL stream, receives change events as they are committed, and publishes them to Kafka via Kafka Connect. Events arrive in Kafka within milliseconds of the database commit, not on the next poll cycle.

The configuration for Debezium’s PostgreSQL connector targeting an outbox table looks roughly like this:

{
  "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
  "database.hostname": "postgres.internal",
  "database.port": "5432",
  "database.user": "debezium",
  "database.dbname": "orders",
  "table.include.list": "public.outbox_events",
  "plugin.name": "pgoutput",
  "transforms": "outbox",
  "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter",
  "transforms.outbox.route.topic.replacement": "${routedByValue}.events"
}

The EventRouter SMT (Single Message Transform) is the key Debezium feature that makes the outbox pattern clean. It reads the standard outbox table columns, aggregatetype, aggregateid, type, and payload, and routes each event to the appropriate Kafka topic based on the aggregate type. An OrderPlaced event for aggregate type Order goes to order.events. A PaymentProcessed event for aggregate type Payment goes to payment.events. The routing is convention-based and zero-configuration if you follow the default column names.

This connects naturally to how Kafka Connect SMTs work in production. The EventRouter is just a transform in the connect pipeline, which means you can chain it with other transforms for payload enrichment, field filtering, or schema evolution handling.

The WAL Retention Problem

CDC introduces an operational consideration that teams consistently underestimate: WAL retention. PostgreSQL will retain WAL segments as long as any replication slot is lagging. If your Debezium connector goes down, your replication slot accumulates lag. PostgreSQL keeps all the WAL segments from the last confirmed LSN forward, and your disk fills up.

In production, I always configure a retention ceiling for the replication slot. In PostgreSQL 13 and later, you can set max_slot_wal_keep_size to cap the WAL retained per slot:

ALTER SYSTEM SET max_slot_wal_keep_size = '10GB';
SELECT pg_reload_conf();

When the WAL retained for a slot exceeds this limit, PostgreSQL invalidates the slot rather than filling the disk. Your Debezium connector will need to restart from scratch via a snapshot, which means re-reading the outbox table, but that is recoverable. Running out of disk on your primary is not recoverable without downtime. Set the limit.

You should also monitor replication slot lag as a first-class SLO. If your Debezium connector is more than a few minutes behind, something is wrong and you need to know before the WAL retention alarm fires.

At-Least-Once Delivery and Idempotent Consumers

This is the constraint teams most often overlook: the outbox pattern gives you at-least-once delivery, not exactly-once. If your relay crashes after publishing a row but before marking it processed, it will publish that row again when it restarts. If your Debezium connector is reset and performs a new snapshot, events may be replayed.

This is not a bug in the pattern. It is the correct trade-off. Exactly-once delivery across database and broker requires either two-phase commit or specialized infrastructure. The outbox pattern gives you the next best thing: no lost events, with the possibility of duplicates, provided your consumers are idempotent.

Making consumers idempotent is a design constraint, not an optional nicety. The most common approach is to include the event ID in each message and track processed event IDs in the consuming service’s database:

CREATE TABLE processed_events (
    event_id UUID PRIMARY KEY,
    processed_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

Before processing any event, the consumer checks whether the event_id exists in processed_events. If it does, skip the event. If it does not, process it and insert the event_id, ideally in the same transaction as the state change. This idempotency table is small and cheap, and it makes your consumer robust to any delivery mechanism. I have written about building idempotency into event-driven architectures on AWS with the same principle, though the outbox pattern applies regardless of which broker you use.

The ACID properties that your database provides are what make both the outbox write and the idempotency check reliable. You are leaning on database atomicity at both ends of the pipeline: once when writing the outbox row, and once when consuming the event.

Connecting the Outbox to CQRS and Event Sourcing

Teams that use CQRS and event sourcing often ask whether they still need the outbox pattern. In a pure event-sourced system, the event store is your source of truth, and your projection consumers read directly from the event log. If you are using a purpose-built event store like EventStoreDB, the delivery guarantee is built into the store. The dual-write problem does not apply in the same way.

But many teams use PostgreSQL as their event store, appending events to an events table and using a read model. In that case, the outbox pattern is exactly the right mechanism for propagating events to other services. The events table is effectively your outbox, and Debezium CDC is the relay.

The two patterns complement each other. Event sourcing answers the question of what happened. The outbox pattern answers the question of how to reliably notify other services that it happened.

Schema Evolution and Payload Versioning

One topic that trips up production deployments is payload schema evolution. When you change the structure of an event payload, old consumers may break on new events, or new consumers may fail on old events that were not yet processed when you deployed.

The standard approach is to version your payload format and include a schema version field in every event:

{
  "schemaVersion": 2,
  "orderId": "...",
  "customerId": "...",
  "lineItems": [...]
}

Consumers should check the schema version before deserializing. If they encounter a version they do not understand, they should dead-letter the message rather than crashing, which gives you time to deploy an updated consumer before re-driving the dead-letter queue.

For teams using Kafka with a schema registry, you can store the Avro or Protobuf schema alongside the event and rely on the registry to enforce compatibility rules. Debezium’s EventRouter SMT supports both JSON and Avro payload formats. This is worth configuring from the start rather than retrofitting when you hit your first breaking change in production.

Choosing Between Polling and CDC

After all of this, the practical decision is: which relay should you start with?

Use polling when you are first adopting the pattern. It is simple to build and simple to operate. It works on any database without any special configuration. The latency is acceptable for most asynchronous workflows. If your team does not already run Kafka Connect infrastructure, polling gives you the full benefit of the outbox pattern without the operational overhead of running connectors.

Move to CDC when your event volume grows high enough that polling queries become a meaningful fraction of your primary’s load, when you need sub-second event propagation, or when you already run Debezium for other data integration purposes. At that point, the EventRouter SMT makes the outbox table a first-class Kafka topic source with minimal additional configuration.

Whichever approach you choose, the application-level changes are identical. The outbox table schema, the transactional insert, and the consumer-side idempotency are the same regardless of whether a polling loop or a WAL follower drives the relay. That separation of concerns is one of the pattern’s practical strengths: you can migrate from polling to CDC without touching your application code.

Production Checklist

Before you ship an outbox-based service to production, verify these items:

Application layer: The outbox table insert happens within the same transaction as the business entity change. There is no try-catch that could commit the business change but swallow an outbox insert failure. Payload serialization happens before the transaction opens so a serialization error does not leave a partial row.

Relay layer: If you use polling, the query uses FOR UPDATE SKIP LOCKED and processes rows in batches with a reasonable limit. If you use CDC, you have set max_slot_wal_keep_size and you are monitoring slot lag. Either way, you have a dead-letter mechanism for events that fail to publish after several retries.

Consumer layer: Every consumer that processes outbox events is idempotent. Event IDs are tracked in a deduplicated store. The idempotency check and the state change are in the same database transaction. The consumer does not assume exactly-once delivery.

Observability: You have metrics for outbox table depth, relay processing lag, and broker delivery latency. You have an alert that fires when outbox depth exceeds a threshold that would indicate the relay is stuck. You have runbooks for what to do if Debezium requires a new snapshot.

Where I Have Seen This Go Wrong

The most common failure mode I encounter is teams who implement the outbox write correctly but forget about idempotency on the consumer side. They ship to production, everything looks fine, and then during a deploy, Debezium restarts and replays events from its last checkpoint. Suddenly the inventory service has processed some orders twice, the email service has sent duplicate confirmations, and the billing service is confused. All of this is avoidable, but only if you treat at-least-once delivery as a given from day one rather than retrofitting idempotency later.

The second failure mode is the WAL retention bomb. A team sets up Debezium, does not configure max_slot_wal_keep_size, and everything runs smoothly until a connector upgrade causes a restart that lags by a few hours. The WAL piles up, the disk fills, and the primary goes read-only. This is a recoverable situation, but it is also a completely preventable one.

The third failure mode is what happens when teams try to use the outbox pattern with a broker that does not have strong ordering guarantees or with consumers that do not use the aggregate ID as the partition key. Two events for the same order can arrive at the consumer out of order. Most business logic is order-sensitive. Use the aggregate ID as the partition key, as the Debezium EventRouter does by default, and this problem disappears.

Wrapping Up

The transactional outbox pattern is mature, battle-tested, and genuinely not complicated once you understand the dual-write problem it solves. The application change is a single extra table and one additional INSERT per transaction. The infrastructure change is either a polling loop or a Debezium connector. The operational discipline required is monitoring relay lag and building idempotent consumers.

The pattern pairs well with change data capture as a relay mechanism, fits naturally into event-driven architectures, and complements Kafka Connect pipelines for teams already running connector infrastructure. If you are running microservices that need to update state and publish events, and you are not using this pattern today, you are accepting reliability risk that the pattern eliminates at low cost.

The dual-write problem is not a corner case. It is the default behavior of any system that writes to two places in sequence. The outbox pattern is the only approach that solves it without introducing distributed transactions, and it has been solving it reliably for years.