294 - Outbox Pattern · dotnetmadeeasy.com

Two separate steps — write to the database, then publish an event — can never be truly atomic. So make it one step, in one database, instead.

Advanced Part VIII's transactions lesson named this pattern before you'd learned what it actually does: "the Outbox pattern... is the modern, more resilient answer to this exact problem for most applications." The problem it was pointing at is one of the most common, deceptively tricky situations in the messaging architecture this Part has been building — a service needs to both update its own database and tell the rest of the system about that change by publishing an event. Two operations. One of them can succeed while the other fails. And the last lesson just showed you why reaching for a full distributed transaction to fix that is expensive, fragile, and often not even possible with the message broker you're actually using.

In this lesson, you'll learn precisely what the "dual write" problem is, why it can't be solved by simply reordering the two steps, and how the Outbox pattern fixes it for real — using nothing more exotic than an ordinary single-database transaction, plus a small background process that does the actual publishing.

What Is It?

The Simple Explanation

The Outbox pattern is a way to guarantee that "update my database" and "tell everyone else about it" either both genuinely happen or neither does — without needing a distributed transaction. The trick: instead of publishing the event to the message broker directly, you write a row describing that event into a plain table in your own database — in the exact same local transaction as the real business change. A separate, simple process later reads that table and does the actual publishing.

The Technical Definition

An outbox table lives in the same database as your business data. When a business operation needs to also announce an event, the operation inserts a row into this table — describing the event's type and payload — as part of the same local database transaction that changes the business data. Because both writes are ordinary rows in the same single database, they get the exact same single-database ACID atomicity you've relied on throughout this course: either both are committed, or neither is. A separate publisher (a polling background worker, or a change-data-capture mechanism reading the database's own transaction log) later reads unpublished rows from the outbox table and actually sends them to the real message broker, marking each as published once the broker confirms receipt.

Why Does It Exist?

The Problem — The Dual Write Isn't Atomic, No Matter the Order

Picture an order service that needs to save a new order to its database and publish an OrderCreatedEvent so the messaging/event-driven architecture covered earlier in this Part can notify inventory, shipping, and notifications. The two operations are naturally written as two separate steps — and separate steps mean there's a real gap between them where things can go wrong:

Database write, then publish

Order saved to the database successfully
Process crashes, or the broker call fails, before the publish completes
Result: the order exists, but nobody downstream ever hears about it — a silently missed event

Publish, then database write

Event published successfully — downstream services start reacting to it immediately
The database write then fails (a constraint violation, a timeout)
Result: downstream services react to an order that, from the source of truth's own perspective, never actually happened

Neither ordering is safe. This is precisely the "dual write" problem — two independent writes to two independent resources, with no way to make "both happen" atomic using ordinary code, because a crash or a failure can always land in the gap between them.

Why Not Just Use a Distributed Transaction?

This is exactly the shape of problem the previous lesson covered — coordinating an atomic outcome across two independent resources (a database and a message broker). And it's exactly the case where 2PC's weaknesses bite hardest: most message brokers don't support participating in a two-phase commit at all, and even where a broker theoretically could, you'd be importing 2PC's blocking, poorly-scaling coordination machinery just to solve what is, underneath, a very common, very ordinary problem. The Outbox pattern sidesteps the whole issue by never actually needing a transaction to span two resources in the first place.

Big Picture

WITHOUT OUTBOX (the dual-write problem)

Business DB write  →  [ GAP — crash/failure risk here ]  →  Publish to broker
     (resource #1)                                            (resource #2)
     Two independent resources. No atomicity between them.


WITH OUTBOX (this lesson's fix)

┌─────────────── ONE local transaction, ONE database ───────────────┐
│  Business DB write   +   Outbox table row insert                  │
│  (ordinary single-database ACID — genuinely atomic, no exceptions) │
└─────────────────────────────────────────────────────────────────────┘
                              │
                              ▼
              Background publisher reads unpublished rows
                              │
                              ▼
                    Publishes to the real message broker
                              │
                              ▼
                 Marks the outbox row as published

The atomicity that used to require coordinating two independent resources is replaced with atomicity between two rows in the same resource — which is exactly the kind of guarantee an ordinary single-database transaction already gives you for free.

How It Works

Simple Example

public class OutboxMessage
{
    public Guid Id { get; set; }
    public string EventType { get; set; } = default!;
    public string Payload { get; set; } = default!;   // serialized event data (e.g. JSON)
    public DateTime CreatedAtUtc { get; set; }
    public DateTime? PublishedAtUtc { get; set; }      // null until actually published
}

public async Task CreateOrderAsync(Order order, CancellationToken ct)
{
    await using var transaction = await _db.Database.BeginTransactionAsync(ct);

    _db.Orders.Add(order);                              // the real business data

    _db.OutboxMessages.Add(new OutboxMessage             // the event, in the SAME transaction
    {
        Id = Guid.NewGuid(),
        EventType = nameof(OrderCreatedEvent),
        Payload = JsonSerializer.Serialize(new OrderCreatedEvent(order.Id, order.CustomerId)),
        CreatedAtUtc = DateTime.UtcNow
    });

    await _db.SaveChangesAsync(ct);   // BOTH rows written together — ordinary ACID atomicity
    await transaction.CommitAsync(ct);
}

Code → Meaning → Result: Nothing here is exotic — it's a single SaveChangesAsync call writing two rows through one DbContext, wrapped in one explicit transaction exactly like the multi-save patterns from Advanced Part VIII. The message broker isn't touched anywhere in this method at all — which is precisely the point: this code can never leave the system in a state where the order exists but the event doesn't, because there's no longer a second resource in the picture at this stage.

public class OutboxPublisherService(AppDbContext db, IMessageBus bus) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            var unpublished = await db.OutboxMessages
                .Where(m => m.PublishedAtUtc == null)
                .OrderBy(m => m.CreatedAtUtc)
                .Take(50)
                .ToListAsync(stoppingToken);

            foreach (var message in unpublished)
            {
                await bus.PublishAsync(message.EventType, message.Payload, stoppingToken);
                message.PublishedAtUtc = DateTime.UtcNow;
                await db.SaveChangesAsync(stoppingToken);   // mark published — if THIS fails, see below
            }

            await Task.Delay(TimeSpan.FromSeconds(2), stoppingToken);
        }
    }
}

This is an ordinary BackgroundService, polling on a short interval — nothing more elaborate than that is required for the pattern to work correctly.

Real-World Example — What Happens When the Publish Step Itself Fails

This is worth walking through concretely, because it's exactly where the previous two lessons in this Part connect directly to this one:

This is precisely why lessons 290 and 291 came before this one, not after: the Outbox pattern guarantees the event will eventually be published at least once — it does not, and cannot, guarantee it will be published exactly once. Idempotent consumers are what make that guarantee good enough in practice.

Analogy

An Actual Outgoing Mail Tray

Picture an office worker finishing a letter. Instead of walking it to the mailbox down the street the instant it's done — risking getting pulled into something else halfway there and forgetting — they drop it in the "outgoing mail" tray on their own desk, right after sealing the envelope. Writing the letter and placing it in the tray happen together, as one uninterrupted action, on the worker's own desk — nothing external is involved yet.

Later, the mailroom clerk makes their rounds, picks up everything sitting in outgoing trays across the office, and actually takes it to the mailbox. If the clerk gets interrupted partway through their rounds, nothing is lost — the letters are still sitting safely in the tray, waiting for the next round. The letter-writer never has to personally guarantee delivery all the way to the mailbox; they only ever had to guarantee the much smaller, much safer act of getting the letter into their own tray. That's the entire trick of the Outbox pattern: shrink the atomicity requirement down to something one system (one database) can actually guarantee on its own, and let a separate, retry-friendly process handle the part that reaches outside.

Under the Hood

Common Confusion

1. "The Outbox pattern replaces the message broker" — no, it sits in front of it

The outbox table isn't a substitute for a real message broker — messages still ultimately flow through the same messaging infrastructure this Part has been building around. The outbox is a temporary, durable staging area inside your own database that solves the specific problem of atomically deciding "this event needs to be published," before handing that job off to the broker exactly as normal.

2. "The Outbox pattern gives exactly-once delivery" — it gives at-least-once, and that's fine, with idempotent consumers

As the Real-World Example walked through directly, a crash between publishing and marking-as-published can cause a genuine duplicate publish. The Outbox pattern's real guarantee is that the event will eventually be published at least once — never zero times, but possibly more than one. That's precisely why this pattern is taught after idempotency and retries in this Part, not before: the guarantee it provides is only as good in practice as the idempotency of whatever consumes it.

Common Mistakes

Mistake 1 — Publishing directly, then writing the outbox row as an afterthought (or vice versa)

Calling the broker first, then writing the outbox row to "record" it — this reintroduces the exact dual-write gap the pattern exists to eliminate, since the two writes are no longer part of the same local transaction the way the pattern requires. The outbox row insert and the business data write must be part of the same database transaction, with the broker never called directly from the business operation at all.

Mistake 2 — Forgetting downstream consumers still need to be idempotent

Assuming the Outbox pattern's careful transactional handling means consumers can skip their own deduplication logic — a duplicate publish, as shown above, is a genuinely possible outcome, not just a theoretical one. Every consumer of an outbox-published event still needs the idempotency techniques from lesson 290 — the outbox pattern and idempotent consumers are a matched pair, not alternatives to each other.

Mistake 3 — Letting the outbox table grow forever

Never cleaning up published rows, so the outbox table grows without bound over the life of the application, slowing down the very query the publisher uses to find unpublished rows. Periodically archive or delete rows that have been marked published and are older than some retention window — the outbox table is meant to be a short-lived staging area, not a permanent event log.

When Should I Use It?

Mental Model

Dual write problem = "update my database" + "publish an event" as two separate, non-atomic operations
Outbox table = write the event as a row, in the SAME transaction as the business data — ordinary single-database ACID does the real work
Background publisher = a separate process that reads unpublished rows and actually sends them to the broker, retrying safely on failure
At-least-once, not exactly-once = a crash between publish and mark-published can cause a duplicate — exactly why downstream consumers must be idempotent

Remember: the pattern doesn't make the dual write atomic across two resources — it eliminates the second resource from the atomic part entirely.

Key Takeaway

The dual write problem — updating a database and publishing an event as two separate operations — has no safe ordering, since a crash or failure can always land in the gap between them.
The Outbox pattern fixes this by writing the event as a row in an outbox table, in the exact same local transaction as the business data change — ordinary single-database ACID makes this genuinely atomic, no distributed transaction required.
A separate background publisher (polling, or change-data-capture) reads unpublished rows and sends them to the real message broker, marking each as published on success.
If the publish step itself fails partway, it simply retries later — the unpublished row is still safely sitting in the outbox table, which is exactly why nothing is ever silently lost.
This guarantee is at-least-once, not exactly-once — a crash between publishing and marking-published can cause a genuine duplicate, which is precisely why every downstream consumer must still be idempotent (lesson 290).
This is the real, standard, modern pattern that has effectively replaced 2PC for the dual-write scenario in production systems today.

Check Your Understanding

You've seen exactly how the Outbox pattern turns a two-resource atomicity problem into a single-database one. Let's confirm it clicked.

1. Why does the Outbox pattern write the event as a row in a database table, rather than publishing directly to the message broker from within the business operation?

A) Writing to a database table is always faster than calling a message broker
B) Because the outbox row can be written in the SAME local transaction as the business data change, giving genuine atomicity via ordinary single-database ACID — something calling an entirely separate broker directly cannot provide
C) Message brokers cannot be called from within a web application's request handling code
D) It's purely a legacy convention with no remaining technical benefit

Show answer

Correct: B

Why B is correct: This is the entire mechanism the pattern relies on — by keeping the event's record inside the same database as the business data, the two writes become genuinely atomic using nothing more than a standard local transaction, sidestepping the dual-write problem entirely.

Why A is incorrect: Speed isn't the reason for this design — atomicity is. A direct broker call might well be faster in isolation; it just can't be made atomic with the database write.

Why C is incorrect: Message brokers can absolutely be called from request handling code — the issue isn't whether it's possible, it's that doing so as a separate step breaks atomicity with the database write.

Why D is incorrect: The pattern is a current, actively used, well-documented solution to a real, ongoing problem — not a legacy leftover.

Reinforcement: The Outbox pattern's power comes entirely from keeping both writes inside one database, where ordinary ACID already applies.

2. The background publisher successfully sends an event to the message broker, but the process crashes before it can update the outbox row's "published" status. What happens on the next polling pass?

A) The row is skipped forever, since it was already sent once
B) The publisher republishes the same event, since from the outbox table's perspective it was never marked as published — this is a legitimate duplicate, which is exactly why downstream consumers need to be idempotent
C) The database automatically detects the broker already received it and prevents republishing
D) The entire outbox table is automatically rolled back to its previous state

Show answer

Correct: B

Why B is correct: This is exactly the scenario the Real-World Example walked through — the publisher has no way to know the broker already received the message, so it retries based on the row's unpublished status, producing a genuine duplicate that idempotent consumers must handle safely.

Why A is incorrect: The row's status column is what the publisher checks — with no "published" mark, it will be picked up again, not skipped.

Why C is incorrect: There's no cross-system mechanism connecting the broker's receipt back to the database automatically — that's precisely the gap that makes the duplicate possible.

Why D is incorrect: Nothing about this scenario triggers any kind of table-wide rollback — the earlier business-data transaction already committed successfully and stays committed.

Reinforcement: The Outbox pattern guarantees at-least-once publishing, not exactly-once — that's why it pairs directly with idempotent consumers.

3. A team implements the Outbox pattern by calling the message broker first, and only afterward writing a row to the outbox table to "log" that the publish happened. What's wrong with this?

A) Nothing — order doesn't matter as long as both eventually happen
B) This reintroduces the original dual-write gap — the broker call and the outbox/business-data write are no longer part of the same atomic local transaction, so a crash between them can still leave the system inconsistent
C) Message brokers reject any message sent before its corresponding outbox row exists
D) This is actually the correct, recommended implementation

Show answer

Correct: B

Why B is correct: The whole point of the pattern is that the outbox row is written in the SAME transaction as the business data, with the broker call happening later, from a separate process. Calling the broker directly from the business operation — before or after the outbox write — recreates exactly the non-atomic dual-write gap the pattern exists to eliminate.

Why A is incorrect: Order matters a great deal here — the correct implementation never calls the broker directly from the business operation at all.

Why C is incorrect: Message brokers have no awareness of an application's outbox table — there's no such enforcement mechanism.

Why D is incorrect: This is precisely Common Mistake 1 from the lesson — the correct approach never calls the broker from inside the business transaction.

Reinforcement: The business operation should only ever write to its own database — publishing to the broker is exclusively the separate background publisher's job.

You now understand exactly how the Outbox pattern solves the dual-write problem using nothing more than ordinary single-database ACID. Next up — the capstone of this Part: eventual consistency, the CAP theorem, and how everything in this Part fits together.