Call it once, or call it five times — an idempotent operation leaves the system in exactly the same state either way.
A customer clicks "Pay Now" once. Somewhere between your payment consumer successfully charging the card and the acknowledgment making it back to the message broker, the network hiccups for half a second. The broker never hears "I handled that one" — so, doing exactly what it's designed to do, it assumes the message was never processed and delivers it again. Your consumer, humming along, picks it up a second time and… charges the card again. The customer paid once. Your system thinks it collected payment twice. Nothing crashed. No bug was thrown. The system worked exactly as every piece of it was designed to — and the customer is still overcharged.
This isn't a rare edge case. It's the routine, expected behavior of the messaging and event-driven systems you've been building throughout this Part. The fix isn't "make delivery more reliable" — it's "make your consumer not care how many times the same message shows up."
In this lesson, you'll learn precisely what idempotency means, why "at-least-once" message delivery makes it a first-class design requirement rather than a nice-to-have, and the concrete techniques — idempotency keys with a deduplication table, and naturally idempotent operations like UPSERTs — that make a consumer safe to call more than once.
An operation is idempotent if doing it once and doing it five times leave the system in the exact same end state. Not "similar." Not "roughly the same." Identical. If you can't tell, just by inspecting the system afterward, whether an operation ran once or ran five times, it's idempotent.
Formally: an operation f is idempotent if applying it n times produces the same result as applying it once — f(f(f(x))) = f(x) for any number of applications. In distributed systems specifically, this is almost always talked about in terms of an operation's effect on state, not its return value: "set the order's status to Shipped" is idempotent because running it 1 time or 100 times leaves the order in the exact same Shipped state. "Increment the shipped count by 1" is not idempotent — running it 100 times increments the count 100 times, and the end state depends entirely on how many times it ran.
The messaging and event-driven architecture covered earlier in this Part let services communicate without a tight, synchronous handshake — a producer publishes a message, and a consumer processes it whenever it gets around to it. But that decoupling comes with an honest trade-off in what guarantee the broker can actually make about delivery. There are, in theory, three possible guarantees:
| Guarantee | What it promises | Reality |
|---|---|---|
| At-most-once | A message is delivered zero or one times — never more | Achievable, but risks silently losing messages on failure |
| Exactly-once | A message is delivered and processed precisely one time, no more, no less | Genuinely difficult to guarantee end-to-end across independent systems; most real messaging infrastructure does not offer true exactly-once delivery semantics for arbitrary consumer-side effects |
| At-least-once | A message is delivered one or more times — it is never silently dropped | The practical, common guarantee real message brokers actually provide |
At-least-once is the pragmatic middle ground: brokers would rather redeliver a message you already handled than risk losing one you never got to. That's the right trade-off for reliability — but it pushes a real obligation onto you, the consumer author: your processing logic must be safe to run more than once on the same message.
It's worth being precise about the mechanism, because it's not a bug in the broker — it's the broker doing its job correctly under an ordinary, unavoidable failure:
Notice: nothing here is a defect. Every component behaved exactly as designed. That's precisely why idempotency can't be an afterthought — it's the only thing standing between "the system worked as designed" and "the customer got charged twice."
WITHOUT IDEMPOTENCY
Message delivered once → charge card → $50 charged
Message redelivered → charge card → $50 charged AGAIN
Total: $100 charged for a $50 order
WITH IDEMPOTENCY (dedup check before applying effects)
Message delivered once → "seen op #8831 before?" NO → charge card → record op #8831 → $50 charged
Message redelivered → "seen op #8831 before?" YES → skip charge, return success
Total: $50 charged, exactly what the customer expects The redelivery itself is not the problem — a redelivered message showing up is completely normal in an at-least-once world. The problem only happens if the consumer reapplies the effect blindly. Idempotency is what breaks that link: the message can arrive again, harmlessly, because the consumer recognizes it's already been handled.
Every message or request carries a unique identifier — an idempotency key (sometimes called a deduplication id, correlation id, or operation id). Before applying the operation's real effects, the consumer checks: "have I already processed this exact id?" If yes, skip straight to returning success without redoing the work. If no, do the work and record the id as processed — atomically, in the same transaction as the business change.
public async Task ProcessPaymentMessageAsync(PaymentMessage message, CancellationToken ct)
{
await using var transaction = await _db.Database.BeginTransactionAsync(ct);
// 1. Have we already processed this exact operation?
bool alreadyProcessed = await _db.ProcessedOperations
.AnyAsync(p => p.IdempotencyKey == message.IdempotencyKey, ct);
if (alreadyProcessed)
{
// Already handled on a previous delivery attempt — do nothing, just succeed.
await transaction.CommitAsync(ct);
return;
}
// 2. Apply the real effect — this is the part that must NEVER run twice.
await _paymentGateway.ChargeAsync(message.CardToken, message.Amount, ct);
// 3. Record that this operation id is now handled — same transaction as the effect.
_db.ProcessedOperations.Add(new ProcessedOperation
{
IdempotencyKey = message.IdempotencyKey,
ProcessedAtUtc = DateTime.UtcNow
});
await _db.SaveChangesAsync(ct);
await transaction.CommitAsync(ct);
}The IdempotencyKey column should carry a unique database constraint, not just an application-level check — that's what closes the race condition where two redelivered copies of the same message are processed concurrently by two different consumer instances (more on this in Common Mistakes below).
Sometimes you don't need a separate dedup table at all, because the operation itself can be phrased in a way that's inherently idempotent — running it 1 time or 100 times produces the same end state, with no bookkeeping required.
-- NOT idempotent: running this twice adds 2 to the count, not 1
UPDATE Orders SET ShippedItemCount = ShippedItemCount + 1 WHERE Id = @orderId;
-- Idempotent: running this once or five times, the end state is identical
UPDATE Orders SET Status = 'Shipped' WHERE Id = @orderId;
-- Idempotent UPSERT: insert if new, update if it already exists — same end state either way
INSERT INTO InventoryReservations (OrderId, Sku, Quantity)
VALUES (@orderId, @sku, @quantity)
ON CONFLICT (OrderId, Sku)
DO UPDATE SET Quantity = EXCLUDED.Quantity;An UPSERT — INSERT ... ON CONFLICT DO UPDATE in PostgreSQL, or a MERGE statement in SQL Server — replaces "blindly insert a new row" (which duplicates on redelivery) with "insert if absent, otherwise overwrite with the same intended final value" (which converges to the same row no matter how many times it runs). This is often the cleanest fix of all, because it needs no separate tracking table — the target row's own key is the deduplication mechanism.
Before and after, on the exact scenario from the hook:
// BEFORE — not idempotent, vulnerable to redelivery
public async Task HandleAsync(ShipOrderMessage message)
{
var order = await _db.Orders.FindAsync(message.OrderId);
order.ShippedCount++; // runs again on redelivery → wrong count
await _emailer.SendShippedEmailAsync(order.CustomerEmail); // sent twice → annoying, or worse
await _db.SaveChangesAsync();
}
// AFTER — idempotent, safe under at-least-once delivery
public async Task HandleAsync(ShipOrderMessage message)
{
var order = await _db.Orders.FindAsync(message.OrderId);
if (order.Status == OrderStatus.Shipped)
return; // already handled on a previous delivery — natural idempotency check
order.Status = OrderStatus.Shipped; // same end state, run once or five times
await _outbox.EnqueueAsync(new OrderShippedEvent(order.Id, message.IdempotencyKey));
await _db.SaveChangesAsync();
}Code → Meaning → Result: The fixed version checks the order's own current status before doing anything — a natural idempotency check with no separate dedup table needed, because "already Shipped" is itself the signal that this message has already been handled. Note the fix also stops sending the email directly from inside the message handler — a consumer-triggered side effect like that should itself go through an idempotency-aware path (the Outbox pattern, covered two lessons from now, is exactly how that's done reliably).
This same discipline shows up outside message queues too, anywhere a caller might legitimately retry a request that could have partially succeeded — a classic case is a checkout API where the client itself might retry a timed-out request:
[HttpPost("charges")]
public async Task<IActionResult> ChargeAsync(
[FromHeader(Name = "Idempotency-Key")] string idempotencyKey,
[FromBody] ChargeRequest request,
CancellationToken ct)
{
// Has this exact idempotency key already produced a result?
var existing = await _db.ChargeResults
.FirstOrDefaultAsync(c => c.IdempotencyKey == idempotencyKey, ct);
if (existing is not null)
return Ok(existing.ToResponse()); // return the SAME result as the first attempt — no re-charge
var chargeId = await _paymentGateway.ChargeAsync(request.CardToken, request.Amount, ct);
_db.ChargeResults.Add(new ChargeResult
{
IdempotencyKey = idempotencyKey, // enforced UNIQUE at the database level
ChargeId = chargeId,
Amount = request.Amount
});
await _db.SaveChangesAsync(ct);
return Ok(new { chargeId });
}This is exactly how real payment providers (Stripe among them) design their public charge APIs — the client generates one idempotency key per logical checkout attempt, and resends the same key on every retry of that same attempt. A dropped connection, a client-side timeout, a mobile app retry after losing signal — none of them risk a double charge, because the server recognizes the repeated key and returns the original result instead of charging again.
Flipping a light switch to "on" is idempotent: flip it once, the room is lit. Flip it four more times (assuming it's a simple on/off switch, not a toggle), and the room is still just lit — the exact same state. You genuinely cannot tell, from the room alone, whether the switch was flipped once or five times.
A tally counter at a museum door is the opposite: every single press adds one, on purpose. If you're trying to count visitors, that's exactly the behavior you want — this operation is supposed to be non-idempotent. The point isn't that idempotent operations are always better; it's that you have to know, precisely, which kind of operation you're building, because treating a tally-counter operation as if it were a light-switch operation — assuming duplicate presses are harmless — is exactly how a duplicate message becomes a duplicate charge.
Charging a card, sending an email, updating a row — these are all real side effects, and an idempotent version of each still has one. The property isn't "no side effect happened"; it's "the side effect's end state is the same whether this ran once or many times." "Set status to Shipped" absolutely changes something in the database — it's idempotent because that change is the same no matter how many times it's applied, not because nothing changed at all.
Idempotency, in the distributed-systems sense used throughout this Part, is about the operation's effect on stored state, not about whether the HTTP response body is byte-for-byte identical across calls. A well-designed idempotent charge endpoint might return a slightly different timestamp field on a duplicate call while still guaranteeing the customer is charged exactly once — the state-level guarantee is what matters, not surface-level response formatting.
Keeping a HashSet<string> of processed ids in memory to check against. This evaporates on a restart, and doesn't work at all once you scale out to multiple consumer instances — each instance has its own, separate set, blind to what the others have seen. The processed-operations record has to live somewhere every consumer instance can see and atomically check against — a shared database table with a unique constraint, not process memory.
A plain SELECT to check, followed later by an INSERT — with no unique constraint and no shared transaction, leaving a real race window under concurrent redelivery (see Under the Hood above). Enforce the uniqueness at the database level and do the effect-plus-record-key write inside one transaction.
Generating a new GUID inside the message handler itself, then using that as the idempotency key — a fresh id every attempt means every redelivery looks "new," defeating the whole point. The idempotency key has to be generated once, by whoever first initiates the operation (the original producer, or the original client request), and carried through unchanged on every retry or redelivery of that same logical operation.
You've seen why at-least-once delivery makes duplicates routine, and the two real techniques for handling them safely. Let's confirm it clicked.
1. Which of the following operations is genuinely idempotent?
Correct: B
Why B is correct: Setting a field to a specific fixed value produces the exact same end state whether it runs once or a hundred times — the row's status is Shipped either way. That's the textbook definition of idempotent.
Why A is incorrect: Incrementing a counter accumulates — running it twice produces a different (higher) end state than running it once. This is the canonical non-idempotent operation.
Why C is incorrect: An unconditional insert with no uniqueness constraint creates a new row every time it runs — two runs leave two rows, a different end state than one run leaving one row.
Why D is incorrect: The customer receiving one email versus five emails is a very different, and noticeably worse, end state — this is not idempotent unless it's specifically guarded with a deduplication check.
Reinforcement: Idempotent operations converge to the same end state regardless of how many times they run — look for "set to a value," not "add to a value."
2. A consumer successfully charges a customer's card, but the acknowledgment back to the message broker is lost due to a network blip. What does the broker do, and why?
Correct: B
Why B is correct: The broker has no visibility into what the consumer actually did — all it knows is whether it received an acknowledgment. With no ack, its only safe assumption under at-least-once delivery is that the message wasn't handled, so it redelivers. This is the exact mechanism the lesson walked through.
Why A is incorrect: Discarding a message with no confirmation is exactly the at-most-once behavior brokers deliberately avoid, because it risks silently losing genuinely unprocessed work.
Why C is incorrect: The broker has no insight into the consumer's internal business logic or external side effects (like a card charge) — it can only track whether it received an acknowledgment message.
Why D is incorrect: This redelivery decision is fully automated based on the acknowledgment timeout — no human is in this loop.
Reinforcement: Duplicate delivery isn't a broker malfunction — it's the broker correctly erring on the side of "redeliver rather than risk losing work," which is exactly why consumers must be idempotent.
3. Why is a database-level unique constraint on the idempotency key column necessary, when the consumer code already checks "have I seen this key before?" as a separate step?
Correct: B
Why B is correct: An application-level "check, then act" sequence has a genuine race window under concurrency — two consumer instances can both check and both see "not yet processed" before either one's write commits. Only a database-enforced uniqueness guarantee reliably prevents both from succeeding.
Why A is incorrect: As Under the Hood explained, the application-level check alone is exactly what leaves the race condition open under concurrent redelivery.
Why C is incorrect: Unique constraints are optional and used specifically to enforce a business rule (in this case, "one row per idempotency key") — they're not a general SQL requirement.
Why D is incorrect: While it can help query performance, the reason it matters here is squarely about correctness — closing a real race condition, not speed.
Reinforcement: True deduplication safety comes from the database's own atomicity guarantee, not from application code alone.
4. A team implements idempotency by generating a brand-new GUID inside the message handler on every invocation, and using that as the "idempotency key" to check against a processed-operations table. What's wrong with this?
Correct: C
Why C is correct: The entire mechanism depends on the SAME key showing up again on a redelivery so the check can recognize it. Generating a fresh id inside the handler itself means every delivery attempt — including duplicates of the exact same underlying operation — gets a different key, so the dedup check will never find a match. This is exactly Common Mistake 3 from the lesson.
Why A is incorrect: The problem isn't the format of the key, it's when and how many times it's generated.
Why B is incorrect: GUID collision is astronomically unlikely and not the issue here at all — the issue is deliberately generating a different key every time, not a collision between two independently-generated ones.
Why D is incorrect: There's no such restriction — string/GUID primary or unique keys are completely standard.
Reinforcement: The idempotency key must originate once, from whoever first initiates the logical operation, and travel unchanged through every retry or redelivery — not be regenerated per attempt.
You now understand precisely why duplicate message delivery is routine, not exceptional — and exactly how to make a consumer safe against it. Next up: retries in the messaging world, and why idempotency is the prerequisite that makes them safe rather than dangerous.
dotnetmadeeasy.com — Learn C# and .NET, the right way.