Your Database Commit Succeeded. Your Message Did Not.

To make the problem concrete, I will use one workflow from an enterprise expense management platform throughout this article. An employee records work expenses and submits a claim for approval. The platform saves the submitted version, places the claim in a review queue, and notifies the reviewer that a decision is waiting. I described the wider system and its transaction boundaries in Four Decisions I Made Before Writing Any Features.

Now imagine that the employee submits the claim. The page confirms it. The claim appears in the review queue. The reviewer never receives the notification.

Nothing is wrong with the claim. The database transaction committed. The missing part is the message that should have left the application afterwards.

I ran into this boundary while working on that platform. One submission had to preserve the business change, record why it happened, and start work outside the request. The first two belonged in PostgreSQL. The last one depended on a worker and an external provider.

That split creates a small but dangerous question:

How do you make a database change and publish a message as one reliable operation when the database and the message provider cannot share a transaction?

The answer I used was a transactional outbox. The idea is simple. Save the business change and a durable description of the message in the same database transaction. A separate worker sends the message later.

The pattern is simple. Making its failure behavior honest is the real work.

The two writes that pretend to be one

Suppose a FastAPI endpoint submits a claim and tells a reviewer about it. A direct implementation might look like this:

async def submit_claim(command: SubmitClaim) -> Claim:
    claim = await claim_service.submit(command)
    await session.commit()

    await notifications.send_review_request(claim.id)
    return claim

There are two separate effects here:

1. PostgreSQL commits the claim.
2. A notification provider accepts the message.

The code places them next to each other. That does not make them atomic.

The process can stop after the commit and before the provider call. The database now says the claim is waiting for review, but no message exists. A retry might notice that the claim is already submitted and return early, so the notification stays lost.

Reversing the order does not help:

await notifications.send_review_request(claim.id)
await session.commit()

Now the reviewer can receive a message about a claim that later rolls back. The link may lead nowhere. Worse, the message can trigger work that no longer has a valid business record behind it.

Holding the database transaction open while making the network call is also not a solution. It keeps locks and connections busy while waiting on a system you do not control. The provider can still accept the message while your connection times out. You are left with the same ambiguous outcome I described in A Timeout Doesn't Tell You Whether the Write Happened, except now the ambiguity is sitting inside a database transaction.

There is no ordering of these two calls that removes the failure window.

database commit ────────┐
                        ├── no shared commit
message provider ───────┘

The useful move is to stop trying to make the external call part of the transaction.

Store the intention before performing the effect

An outbox turns “send this message” into data that PostgreSQL can commit.

The submission transaction writes three things:

• the new business state;
• the audit event that explains the change;
• an outbox event that records the work to be done.

Either all three rows commit, or none of them do.

async def submit_claim(command: SubmitClaim) -> Claim:
    async with session.begin():
        claim = await claim_service.submit(command)

        session.add(
            AuditEvent(
                action="claim.submitted",
                object_id=claim.id,
                actor_id=command.actor_id,
            )
        )

        session.add(
            OutboxEvent(
                event_type="claim.submitted.v1",
                aggregate_id=claim.id,
                payload={"claim_id": str(claim.id)},
            )
        )

    return claim

The important detail here is the transaction boundary. The repositories used by the operation may add and flush rows, but they do not commit on their own. The application service owns one explicit commit.

The outbox is what makes the last of those architectural decisions operational. The notification intent cannot disappear without the claim submission disappearing with it.

Notice what the transaction does not contain. It does not send email. It does not publish to a broker. It does not call a webhook. It records enough durable state for those things to happen later.

That is the core promise of the pattern:

The outbox does not guarantee that an external effect has happened. It guarantees that the application will not forget that the effect still needs to happen.

The event needs a real contract

An outbox row is not a place to dump an ORM object. It is a message contract.

A useful minimal shape looks like this:

id
event_type
event_version
aggregate_id
payload
status
available_at
attempt_count
lease_owner
lease_expires_at
created_at

id gives the logical event a stable identity. event_type and event_version tell a consumer how to interpret it. aggregate_id lets the system preserve or reason about ordering for one business object. available_at supports delayed retries. The lease fields coordinate workers.

The payload should be small and versioned. It should contain what the consumer needs, not every column the application happened to have loaded. Large snapshots make migrations harder and increase the chance of copying sensitive data into another system.

There are two sensible payload styles.

The first is a notification payload. It contains enough facts for the consumer to act without reading the source database. This reduces coupling, but the event becomes a durable copy of those facts.

The second is a reference payload. It contains identifiers, and the consumer reads current data before acting. This keeps the event small, but it couples delivery to the availability and current meaning of the source record.

Neither is always right. For an email that must reflect the exact submitted version, I prefer a bounded snapshot or an exact version reference. A pointer to “the current claim” can silently change meaning before the worker reads it.

Claim the work, then release the transaction

The worker has a different transaction boundary from the command.

It first claims a small batch of ready events. With PostgreSQL, a common approach is FOR UPDATE SKIP LOCKED:

WITH candidates AS (
    SELECT id
    FROM outbox_events
    WHERE status = 'READY'
      AND available_at <= now()
    ORDER BY available_at, id
    FOR UPDATE SKIP LOCKED
    LIMIT :batch_size
)
UPDATE outbox_events AS event
SET status = 'LEASED',
    lease_owner = :worker_id,
    lease_expires_at = now() + :lease_duration
FROM candidates
WHERE event.id = candidates.id
RETURNING event.*;

Several workers can run this query at the same time. A row locked by one worker is skipped by the others. PostgreSQL's own documentation warns that SKIP LOCKED gives an inconsistent view and is unsuitable for normal reads. It explicitly calls out queue-like tables as the useful case. An outbox is one of those cases. See the PostgreSQL locking-clause documentation.

The worker then commits the lease before making a network call.

events = await outbox.claim_batch(worker_id)

for event in events:
    try:
        await dispatch(event)  # no SQL transaction held here
    except RetryableProviderError as error:
        await outbox.schedule_retry(event, worker_id, error)
    except TerminalDeliveryError as error:
        await outbox.mark_dead(event, worker_id, error)
    else:
        await outbox.mark_delivered(event, worker_id)

The provider call happens outside the claim transaction. This keeps database locks short. It also means a slow provider does not consume a database connection for the whole wait.

Every final write should check ownership. A worker whose lease has expired must not be able to mark an event delivered after another worker has reclaimed it. The update should include the event ID, current status, lease owner, and expected lease state. If no row matches, the worker no longer owns the decision.

This is the same reason a background job needs ownership-safe completion. A lease is not a label. It is permission that expires.

The lease must cover the bounded dispatch window or be renewed while the worker is still making progress. Even then, it cannot prevent every duplicate provider call. A paused worker can resume after another worker has reclaimed the event. Provider-side idempotency is still required.

The outbox gives you at-least-once work

The hardest failure happens after the provider accepts the message but before the worker records success.

worker                     provider                 database
  |                            |                        |
  |  send event -------------->|                        |
  |<--------- accepted --------|                        |
  |                            |                        |
  X process stops              |                        |
                               |                        |  still LEASED/READY

When the lease expires, another worker retries the event. The outbox did its job: it did not lose the work. The provider may now receive the same logical message twice.

This is why “transactional outbox gives exactly-once delivery” is wrong. It usually gives at-least-once processing. Duplicate delivery is part of the contract.

The receiver needs a stable idempotency key. The outbox event ID is a good candidate. If the provider supports idempotency keys, send it with every attempt. If you own the consumer, store processed event IDs under a unique constraint and make applying the event and recording that ID one transaction.

async with consumer_session.begin():
    if await processed_events.exists(event.id):
        return

    await apply_event(event)
    await processed_events.add(event.id)

The unique constraint is still necessary. Two copies can arrive together. “Check, then insert” without a database invariant has a race in it.

If the destination cannot deduplicate and the effect cannot safely repeat, the worker needs a reconciliation strategy. That may be a lookup, a provider receipt, or a manual uncertain state. The outbox cannot make an unsafe external API safe by itself.

Ordering is a separate decision

Once multiple workers skip locked rows, strict global ordering is gone. That is often fine. A password-reset email and a report-generation request do not need one shared order.

Ordering can matter within one aggregate. A consumer should not see claim.approved.v1 before claim.submitted.v1 for the same claim.

There are several ways to handle this:

• include an aggregate sequence and reject or defer gaps;
• route one aggregate to one broker partition;
• prevent a later event for the same aggregate from being leased while an earlier one is unfinished;
• design consumers around current durable state instead of assuming delivery order.

Each option has a cost. Global serialization is the most expensive and is rarely needed. State the ordering guarantee you actually require, then enforce that guarantee only.

Debezium provides another implementation path. Its Outbox Event Router captures committed outbox rows and routes them to a broker. That removes the polling publisher from application code. It does not remove the need for stable event IDs, consumer idempotency, ordering decisions, or operational monitoring.

A dead event should stay dead

Some failures are transient. A provider is unavailable, a connection resets, or a rate limit asks you to return later. These can retry with bounded exponential backoff and jitter.

Other failures are terminal. The event version is unsupported. The destination no longer exists. The payload violates a contract. Repeating those calls only creates noise.

After a bounded number of attempts, the event should enter a dead state. Preserve its payload, attempt history, and final error classification. Do not turn replay into “set attempts to zero and pretend this is new.”

I prefer replay to create a new event linked to the dead original. That keeps both facts:

• the original event failed after its recorded attempts;
• an authorized person later asked the system to try again under a new event identity.

This is a small audit decision with a large operational benefit. Nobody has to guess whether an old row was edited after the incident.

Monitor delay, not only failure count

An outbox can have zero dead events and still be failing users.

If the worker stopped five minutes ago, every event may still be valid and ready. A dashboard that counts only terminal failures stays green while the queue grows.

The useful signals are:

• age of the oldest ready event;
• number of ready, leased, retrying, and dead events;
• leases that expired without completion;
• attempts by event type and error class;
• delivery duration and provider response class;
• events created versus events completed over time.

The oldest-ready age is often the clearest user-facing measure. It answers, “How late is the work we promised to do?” A raw queue count cannot answer that. Ten recent events may be healthy. One event waiting for an hour may not be.

The worker also needs a safe shutdown path. It should stop taking new leases, finish or cancel bounded active work, and leave incomplete events recoverable after their leases expire.

Do not route every background task through it

Once an outbox exists, it is tempting to use it for every scheduled or deferred operation. I avoid that.

The pattern earns its complexity when a database transaction creates an obligation to perform work outside that transaction. Sending a notification, publishing an integration event, and requesting an external document scan fit.

A periodic task that only examines and updates the same database may not need an outbox. It can run directly with its own locking, idempotency, and transaction rules. Adding an event in front of it can create a second state machine without solving a real dual-write problem.

Use the outbox at an external-effect boundary, not as a synonym for “background job.”

What the pattern does and does not solve

A transactional outbox gives you a durable handoff between business state and later work. It closes the gap where the database commits but the application forgets to publish.

It does not give you exactly-once delivery. It does not make an unsafe provider idempotent. It does not choose your ordering model. It does not classify errors for you. It does not monitor itself.

The full design is a set of smaller promises:

• PostgreSQL atomically stores the business change and the intent.
• A lease makes one worker the temporary owner of an attempt.
• Network work happens outside database transactions.
• Ownership checks stop stale workers from finalizing.
• Stable event IDs make duplicate delivery manageable.
• Bounded retries and immutable dead letters keep failure visible.
• Backlog age tells operators whether the handoff is keeping its promise.

That is what made the pattern useful to me. It did not make the database and the provider share one commit. It made the boundary between them explicit, durable, and recoverable.

The database commit succeeded. The message did not. Now the system knows that those are two different facts.