291 - Retries · dotnetmadeeasy.com

Retrying a failed HTTP call and retrying a failed message are the same idea wearing very different clothes — and without idempotency, retrying isn't a safety net, it's a hazard.

Advanced Part VII's resilience lesson taught you to retry a failing HttpClient call — a synchronous request-response exchange, where the caller is sitting right there, waiting, and decides for itself whether to try again. Message processing is a different shape entirely: a consumer pulls a message off a queue, and if handling it throws, the question of "should this be retried?" isn't really the consumer's private decision anymore — it's a negotiation between the consumer and the broker, governed by retry limits, redelivery counts, and, eventually, a dead-letter queue for messages nothing can rescue.

And here's the part that makes this genuinely more dangerous than the HTTP case: a synchronous caller retrying a timed-out request is usually only retrying once, from one place. A message-based retry can happen automatically, repeatedly, driven entirely by the broker — and if the operation being retried isn't safe to run twice, every one of those automatic retries is a live risk of duplicating its effect. That's not a hypothetical: it's precisely why idempotency, the lesson right before this one, isn't optional background reading — it's the thing that makes everything in this lesson safe to use at all.

In this lesson, you'll learn how retrying failed message processing actually works — retry limits, redelivery counts, and dead-letter queues as the endpoint for exhausted retries — why idempotency is a genuine prerequisite for safe retries in this world, and how exponential backoff with jitter (already covered in Part VII) applies to a whole fleet of consumer instances retrying against the same struggling dependency.

What Is It?

The Simple Explanation

A message retry is what happens when a consumer fails to process a message — an exception is thrown, a downstream call times out, a transient database error occurs — and instead of giving up immediately, the message is made available to be processed again, either by the same consumer or a different instance of it.

The Technical Definition — How This Differs From an HTTP Retry

In the HTTP case from Part VII, the retrying party and the calling party were the same code, in the same call stack, deciding synchronously and immediately whether to try again. In message processing, the retry is mediated by the broker itself: when a consumer fails to acknowledge a message within its processing window (or explicitly signals failure), the broker makes that message visible again for redelivery — tracking a redelivery/delivery-attempt count on the message itself, and, once that count crosses a configured limit, routing the message to a dead-letter queue instead of retrying it again.

HTTP Retry (Part VII)

Caller decides, synchronously, whether to retry
Retry happens in the same call, same process
Caller directly controls the retry policy
Failure means the whole operation fails right now

Message Retry (this lesson)

Broker decides when to make the message available again
Retry may be picked up by a different consumer instance
Retry count and limits are tracked on the message/subscription, not in your code alone
Exhausted retries route to a dead-letter queue, not an immediate caller-visible failure

Why Does It Exist?

The Problem — Message Handling Fails for the Same Reasons Any Distributed Call Does

A consumer processing a message often does real work against real dependencies — a database write, a call to another service, an external API. Every one of those can fail transiently, for exactly the reasons Part VII already covered: a brief network blip, a downstream service under momentary load, a deadlock retried at the database layer. None of that is unique to messaging. What is different is who's watching and what happens next: there's no synchronous caller sitting on the line waiting for an answer, so the retry decision has to be made by infrastructure — the broker — rather than by a caller who's already moved on.

The Solution — Bounded Retries, Then a Dead-Letter Queue

Retrying forever isn't a solution — a message that can never be processed (a permanently malformed payload, a business rule that will never pass) would loop indefinitely, consuming processing capacity for no benefit. The standard answer is a retry limit: attempt processing some bounded number of times, and if every attempt fails, stop retrying and move the message somewhere a human or a separate process can look at it — the dead-letter queue, a concept the messaging lessons earlier in this Part already introduced. This turns "endlessly stuck, consuming resources" into "safely parked, visible, and out of the way of everything else still flowing through the system."

Big Picture

Message delivered to consumer
    ↓
Processing throws / times out
    ↓
Broker sees no successful acknowledgment
    ↓
Delivery count incremented, message becomes available again
    ↓
Delivery count < max retries?  ──YES──▶  Redeliver (with backoff + jitter) ──▶ back to "Processing"
    │
    NO
    ↓
Route to Dead-Letter Queue — stop retrying, park it for inspection

This loop is exactly why the immediately preceding lesson matters here: every pass back through "Processing" is, from the operation's point of view, a duplicate delivery of the same underlying work — precisely the scenario idempotency exists to make safe.

How It Works

Retry Limits and Delivery Counts

Real message brokers attach a delivery-attempt counter to each message (SQS calls it ApproximateReceiveCount, Azure Service Bus calls it DeliveryCount — the exact name varies, the mechanism doesn't). Every time a message is delivered without being successfully acknowledged, this counter increments. A configured max delivery count — say, 5 — caps how many times the broker will keep trying before giving up on normal delivery entirely.

Dead-Letter Queues as the Retry Endpoint

Once a message's delivery count exceeds the configured maximum, the broker stops offering it for normal processing and instead routes it to a separate destination — the dead-letter queue, which the messaging lessons earlier in this Part already introduced as the holding area for messages that couldn't be handled. For the purposes of this lesson, the important connection is simply this: the dead-letter queue is the literal, designed endpoint that a bounded retry policy reaches when it gives up — it's not a failure of the retry mechanism, it's the retry mechanism's planned, correct final outcome for a message that genuinely cannot be processed.

Exponential Backoff and Jitter, Applied to a Fleet of Consumers

Part VII already covered exponential backoff with jitter thoroughly — each retry waits progressively longer, with randomized variation, specifically to avoid every client retrying at the exact same instant. The mechanism doesn't change here; the scale of what it's protecting does. A single misbehaving HttpClient caller thundering-herding a dependency is one problem. A consumer group with twenty scaled-out instances, all processing messages against the same struggling downstream service, all retrying failed messages at the same fixed interval, is the exact same thundering-herd mechanism — just multiplied by however many consumer instances you're running. Backoff with jitter is what keeps twenty simultaneous retries from landing on a recovering dependency in one synchronized wave, exactly as it did for HTTP calls in Part VII.

Simple Example

A message handler that combines a bounded retry with the idempotency check from the previous lesson — notice how little new machinery is actually needed once the idempotency check is already in place:

public async Task HandleAsync(ChargeOrderMessage message, MessageContext ctx, CancellationToken ct)
{
    // Idempotency guard from lesson 290 — makes every retry below safe by construction
    if (await _db.ProcessedOperations.AnyAsync(p => p.IdempotencyKey == message.IdempotencyKey, ct))
    {
        await ctx.CompleteMessageAsync(ct); // already handled — acknowledge and move on
        return;
    }

    try
    {
        await _paymentGateway.ChargeAsync(message.CardToken, message.Amount, ct);
        await RecordProcessedAsync(message.IdempotencyKey, ct);
        await ctx.CompleteMessageAsync(ct);          // success — tell the broker not to redeliver
    }
    catch (TransientGatewayException)
    {
        // Don't complete the message — let the broker's own delivery-count + backoff
        // policy decide whether to redeliver. After N failed attempts, the broker
        // routes this message to the dead-letter queue automatically.
        throw;
    }
}

Code → Meaning → Result: The handler doesn't implement its own retry loop at all — it leans on the broker's built-in delivery-count tracking and backoff configuration, which is the standard shape for message-based retries (as opposed to the HTTP case, where your own code typically owns the retry loop directly via a resilience pipeline). What the handler does own is the idempotency check — because no amount of broker-side retry configuration makes a non-idempotent charge safe to repeat.

Real-World Example — Inventory Reservation Under Load

An order-fulfillment consumer reserves inventory for each line item by calling an internal inventory service. Under a traffic spike, that inventory service starts timing out for a few seconds — a completely ordinary, transient blip, but one hitting every one of the fulfillment consumer's ten scaled-out instances at once:

Without idempotency + backoff	With idempotency + backoff (this lesson's approach)
Reservation call times out after partially succeeding server-side; retry blindly re-reserves the same stock, over-reserving inventory that doesn't exist	Reservation is keyed by an idempotency key per order line; retry recognizes the prior attempt and returns the existing reservation instead of creating a duplicate one
All ten consumer instances retry at the same fixed interval, slamming the already-struggling inventory service with a synchronized wave right as it tries to recover	Each instance's retry timing is staggered by exponential backoff with jitter, spreading the retry load out instead of piling on at once
Messages that can never succeed (e.g. a SKU that no longer exists) retry forever, consuming processing capacity indefinitely	After the configured max delivery count, the message routes to the dead-letter queue, where it's visible for a human to investigate instead of looping silently

Under the Hood

Common Confusion

"Retries make my system more reliable" — only if what's being retried is safe to repeat

It's tempting to treat "add retries" as an unconditionally good move — more attempts, more chances to succeed, sounds strictly better. But a retry only improves reliability if repeating the operation is harmless (or actively safe, via idempotency) when it happens to succeed on an attempt after a prior attempt actually already succeeded server-side. Without that guarantee, "more attempts" literally means "more chances to duplicate an effect that already happened" — the exact opposite of what you were trying to achieve.

"A dead-lettered message is a bug" — sometimes, but not always

It's easy to treat every dead-lettered message as an incident to panic over. Some genuinely are (a downstream outage that outlasted the retry window). But some are entirely expected — a message with permanently invalid data that will never successfully process no matter how many times it's retried. The dead-letter queue's job is simply to stop that message from looping forever and make it visible for a human or automated process to decide what to do next — landing there is the retry system working correctly, not failing.

Common Mistakes

Mistake 1 — Retrying a non-idempotent operation with no dedup protection

Configuring a generous retry policy on a message handler that charges a card, with no idempotency key or dedup check anywhere in the path — every redelivery is a live risk of a duplicate charge. Idempotency isn't optional polish here; it's the precondition that makes turning retries on safe in the first place.

Mistake 2 — No retry limit at all, or one set far too high

Leaving max delivery count unset or absurdly high, so a permanently unprocessable message (bad data, a business rule that will never pass) loops for hours, tying up processing capacity that healthy messages need. Set a deliberate, bounded retry limit and let genuinely exhausted messages land in the dead-letter queue where they're visible, instead of silently consuming resources forever.

Mistake 3 — Fixed, un-staggered retry delays across a scaled-out consumer group

Every consumer instance retries a failed message after exactly the same fixed delay — under real scale-out, this reproduces the exact thundering-herd problem Part VII covered for HTTP calls, just aimed at whatever downstream dependency the message handler calls into. Configure (or implement, where the broker's own retry policy doesn't cover it) exponential backoff with jitter, exactly as already learned — the mechanism doesn't change, only the context it's applied in.

When Should I Use It?

Mental Model

Message retry = broker-mediated redelivery, tracked by a delivery count, bounded by a max-attempts policy
Dead-letter queue = where bounded retries end up once they're exhausted — the planned endpoint, not a malfunction
Idempotency = the precondition that makes retrying safe at all — without it, retries duplicate effects instead of just recovering from blips
Backoff + jitter = the same mechanism from Part VII, now protecting a downstream dependency from an entire fleet of retrying consumer instances at once

Remember: a retry policy is only as trustworthy as the idempotency guarantee underneath it.

Key Takeaway

Message retries are mediated by the broker, tracked via a delivery/redelivery count, and bounded by a configured max-attempts limit.
A dead-letter queue is the designed endpoint a message reaches once it exhausts its retries — a normal, expected outcome for a genuinely unprocessable message, not necessarily a bug.
Idempotency is a genuine prerequisite for safe retries — if reprocessing has a different effect than processing once, retries stop being a reliability mechanism and become a correctness hazard.
Exponential backoff with jitter, already covered for HTTP calls in Part VII, applies the same way here — protecting a downstream dependency from a synchronized wave of retries across an entire scaled-out consumer group.
Combine the two: idempotency makes retries safe, backoff-with-jitter makes them considerate of the systems they're retrying against.

Check Your Understanding

You've seen how message retries differ from HTTP retries, and why idempotency is what makes them trustworthy. Let's confirm it clicked.

1. A message handler charges a customer's card, but a database write immediately afterward times out. The broker, seeing no acknowledgment, redelivers the message, and the handler runs again with no idempotency protection in place. What happens?

A) The broker automatically detects the card was already charged and skips the charge on redelivery
B) The card gets charged a second time — the retry mechanism, meant to improve reliability, has instead duplicated a real-world effect
C) The message is automatically discarded without being retried
D) Nothing — retries never re-execute code that already ran once

Show answer

Correct: B

Why B is correct: With no idempotency protection, the handler has no way to recognize this is a redelivery of an operation it already completed — it simply runs the charge logic again, producing a genuine duplicate charge. This is exactly the danger the lesson centers on.

Why A is incorrect: The broker has no visibility into the consumer's business logic or external side effects — it only tracks delivery/acknowledgment, not what the handler actually did.

Why C is incorrect: Under at-least-once delivery with no successful acknowledgment, the broker's default behavior is to redeliver, not discard.

Why D is incorrect: A redelivered message runs the handler's code again from the top — there's no automatic "already ran" protection unless the handler itself implements it.

Reinforcement: Without idempotency, a retry is not a safe do-over — it's a full, unguarded re-execution of the operation's real effects.

2. What is the relationship between idempotency and retries in the distributed-messaging context this lesson covers?

A) They are unrelated — idempotency only matters for HTTP APIs, not message processing
B) Idempotency is a genuine prerequisite for safe retries — without it, retrying a message that already succeeded can duplicate its effect instead of merely recovering from a transient failure
C) Retries make idempotency unnecessary, since the broker handles deduplication automatically
D) Idempotency and retries solve the exact same problem and are interchangeable

Show answer

Correct: B

Why B is correct: This is the lesson's central point: a retry only improves reliability if reprocessing is safe. Idempotency is exactly what makes reprocessing safe — without it, retries risk duplicating effects rather than recovering from blips.

Why A is incorrect: Idempotency matters just as much — arguably more — in message processing, since redelivery there is automatic and broker-driven rather than a single caller-initiated retry.

Why C is incorrect: Most brokers do not automatically deduplicate based on business-level operation identity — that's exactly why the application has to implement its own idempotency check (lesson 290).

Why D is incorrect: They're complementary, not the same thing — retries decide whether to try again; idempotency decides whether trying again is safe.

Reinforcement: Retries and idempotency are two halves of the same safety mechanism — one without the other is incomplete.

3. A message has been redelivered and has failed processing five times, hitting the configured max delivery count. What is the expected, correct outcome?

A) The broker keeps retrying indefinitely regardless of the configured limit
B) The message is permanently and silently deleted with no trace
C) The message is routed to the dead-letter queue — the designed endpoint for a message that has exhausted its retries, making it visible for investigation instead of looping forever
D) The entire message queue is paused until an administrator manually intervenes

Show answer

Correct: C

Why C is correct: This is the standard, designed behavior of a bounded retry policy — once the max delivery count is reached, the message is moved to the dead-letter queue rather than continuing to retry or being lost entirely.

Why A is incorrect: The whole point of a configured max delivery count is to stop retries at that limit, not ignore it.

Why B is incorrect: Silently deleting the message would make failures invisible — the dead-letter queue exists specifically to avoid this by keeping the message visible.

Why D is incorrect: Other unrelated messages continue flowing normally — only the specific exhausted message is set aside; the whole queue does not halt.

Reinforcement: A dead-lettered message is the retry system succeeding at its actual job — bounding retries and surfacing what couldn't be resolved automatically.

4. Ten scaled-out instances of the same consumer are all retrying failed messages against the same struggling downstream service, each using an identical fixed retry delay with no jitter. What problem does this most closely resemble from Part VII's resilience lesson?

A) A deadlock between the ten consumer instances
B) A thundering herd — all ten instances retry in a synchronized wave, potentially overwhelming the downstream service right as it's trying to recover, exactly as fixed-delay HTTP retries did in Part VII
C) A memory leak in the message broker
D) This scenario has no equivalent in the earlier resilience lesson

Show answer

Correct: B

Why B is correct: This is the exact same thundering-herd mechanism Part VII covered for HTTP retries, just scaled up to a whole fleet of message consumers instead of individual HTTP callers — synchronized fixed-delay retries can pile onto a recovering dependency all at once.

Why A is incorrect: There's no mutual resource contention between the consumer instances themselves described here — the issue is coordinated retry timing against a shared downstream dependency, not a deadlock.

Why C is incorrect: Nothing in the scenario points to memory growth in the broker — the issue is retry timing and downstream load.

Why D is incorrect: This is explicitly the same mechanism as Part VII's HTTP thundering-herd scenario, just applied to message consumers — the lesson draws this connection directly.

Reinforcement: Exponential backoff with jitter is the same fix here as it was for HTTP retries — spread retries out so they don't land on the dependency in one synchronized wave.

You now understand how message retries actually work, and why idempotency isn't optional groundwork — it's the thing that makes retries trustworthy. Next up: circuit breakers, the full treatment Part VII promised.