267 - Resilience & Fault Handling

A dependency that fails once is a blip. A thousand clients retrying it at the exact same instant is an outage you caused.

Your API calls a third-party shipping-rate service on every checkout. One afternoon, that service hiccups for four seconds under its own load spike — a completely normal, survivable blip for a well-behaved client to just retry past. Except every one of your app's in-flight requests notices the failure at roughly the same moment, and every one of them retries after exactly the same fixed one-second delay you hardcoded. A wall of retries lands on the shipping service at the exact same instant, right as it's trying to recover — and that wave of simultaneous retries is often enough to knock it back down, or keep it down longer than the original blip ever would have. You didn't just fail to handle a transient fault. You made it worse.

In this lesson, you'll learn the three pillars of resilient calls to external dependencies — retries (done correctly, with exponential backoff and jitter), circuit breakers, and timeouts — and how to wire all three into IHttpClientFactory using Microsoft.Extensions.Resilience.

What Is It?

The Simple Explanation

Resilience is the practice of assuming, up front, that the external things your code depends on — other APIs, databases, third-party services — will occasionally fail, slow down, or become temporarily unreachable, and writing your code so that when that happens, it degrades gracefully instead of falling over or making the problem worse.

The Technical Definition

Modern .NET builds resilience on top of Polly, the long-established .NET resilience and transient-fault-handling library, via the Microsoft.Extensions.Resilience package. Since .NET 8, this integrates directly into IHttpClientFactory — the typed/named client infrastructure you already know from the Intermediate course — through AddResilienceHandler(...), letting you attach a named pipeline of resilience strategies (retry, circuit breaker, timeout, and others) to any HttpClient registration with a few lines of configuration.

Why Does It Exist?

The Problem — Networks and Dependencies Fail, Constantly, Briefly

Once your app makes real network calls — to a database, a third-party API, another internal service — it inherits every way a network call can go wrong: a momentary DNS hiccup, a server briefly overloaded, a load balancer mid-deployment, a packet dropped somewhere in between. Most of these are transient — gone in milliseconds to seconds, not permanent outages. Code that treats every failure as fatal (bubble the exception straight up, fail the whole request) throws away perfectly recoverable situations. Code that retries naively — same delay, every time, for every client — can turn one dependency's bad five seconds into everyone's bad five minutes, through the exact thundering-herd mechanism in the hook above.

The Solution — Assume Failure, Handle It Deliberately

Resilience isn't one trick — it's a small set of complementary strategies, each solving a different failure shape: retries recover from brief blips, circuit breakers stop hammering something that's clearly down, and timeouts stop you from waiting forever on something that will never answer. Used together, they let your app survive the kind of failures that are completely normal in any system with real network dependencies.

Big Picture

Your code
    ↓
Timeout        — "never wait forever for an answer"
    ↓
Retry (with backoff + jitter) — "a brief blip? try again, spaced out sensibly"
    ↓
Circuit Breaker — "clearly and repeatedly failing? stop hammering it, fail fast for a while"
    ↓
The external dependency (API, database, third-party service)

These three wrap around a single outbound call, each addressing a different question: how long do I wait (timeout), do I try again on failure and how (retry), and should I even bother trying right now given recent history (circuit breaker)?

How It Works

Retries — And the Thundering Herd Problem

WHY A FIXED RETRY DELAY IS DANGEROUS AT SCALE

1. A DEPENDENCY BRIEFLY FAILS FOR EVERYONE AT ONCE

Say a downstream service has a 2-second blip. Every client currently calling it — potentially hundreds or thousands of concurrent requests — sees a failure at roughly the same moment.

▼

2. WITH A FIXED, IDENTICAL RETRY DELAY, THEY ALL RETRY IN LOCKSTEP

If every client's rule is "wait exactly 1 second, then retry," they all retry within the same narrow window — producing a sudden, synchronized spike of load exactly when the dependency is trying to recover. This spike, the thundering herd, can be worse than the original failure and can even re-trigger it.

▼

3. THE FIX — EXPONENTIAL BACKOFF WITH JITTER

Exponential backoff: each successive retry waits progressively longer than the last (e.g. roughly 1s, then 2s, then 4s) — later attempts back off further, giving the dependency real breathing room to recover instead of being hit again almost immediately.
Jitter: a small, randomized amount of variation is added to each client's wait time, so instead of every client retrying at exactly "2 seconds," one retries at 1.8s, another at 2.3s, another at 2.1s — the retries spread out across a window instead of landing in one synchronized spike.

Circuit Breakers — Failing Fast Instead of Piling Up

CIRCUIT BREAKER STATES (INTRODUCTORY LEVEL)

1. CLOSED — NORMAL OPERATION

Requests flow through to the dependency normally. The circuit breaker is quietly counting recent failures.

▼

2. OPEN — AFTER ENOUGH CONSECUTIVE FAILURES

Once failures cross a configured threshold, the circuit "opens." For a set period, calls to that dependency fail immediately, without even attempting the network call — this is "failing fast" instead of piling up more slow, doomed requests behind an already-struggling dependency.

▼

3. HALF-OPEN — TESTING RECOVERY

After the open period elapses, the breaker allows a limited number of test requests through. If they succeed, the circuit closes again (back to normal); if they still fail, it re-opens and waits again.

Scoped intentionally: This is an introductory, conceptual treatment of circuit breakers — enough to recognize the pattern, configure a basic one, and understand why it exists. A fuller, deeper treatment of circuit breakers as part of broader distributed-systems resilience may appear in a later, more distributed-systems-focused part of this course.

Timeouts — Never Wait Forever

A call with no timeout can, in the worst case, hang indefinitely — tying up a thread, a connection, and the caller's own responsiveness while waiting for an answer that may never come. A timeout guarantees the call gives up after a bounded, known amount of time, converting an indefinite hang into a definite, handleable failure.

Simple Example

A single retry strategy, attached to a named HttpClient via AddResilienceHandler:

builder.Services.AddHttpClient("ShippingApi", client =>
{
    client.BaseAddress = new Uri("https://shipping.example.com/");
})
.AddResilienceHandler("shipping-retry", pipeline =>
{
    pipeline.AddRetry(new HttpRetryStrategyOptions
    {
        MaxRetryAttempts = 3,
        BackoffType = DelayBackoffType.Exponential,
        UseJitter = true,                         // spreads retries out — avoids thundering herd
        Delay = TimeSpan.FromSeconds(1)            // base delay before exponential growth + jitter
    });
});

BackoffType = DelayBackoffType.Exponential makes each retry wait longer than the last; UseJitter = true adds the randomized variation that spreads many clients' retries across a window instead of one synchronized spike — this single flag is the direct fix for the thundering-herd scenario from the hook.

Real-World Example

A payment-gateway typed client, combining retry, circuit breaker, and timeout into one resilience pipeline — the standard shape for a real production integration with an external dependency:

public class PaymentGatewayClient(HttpClient httpClient)
{
    public async Task<bool> ChargeAsync(string cardToken, decimal amount, CancellationToken ct)
    {
        var json = $$"""{"token": "{{cardToken}}", "amount": {{amount}}}""";
        var content = new StringContent(json, System.Text.Encoding.UTF8, "application/json");

        var response = await httpClient.PostAsync("charges", content, ct);
        return response.IsSuccessStatusCode;
    }
}

builder.Services.AddHttpClient<PaymentGatewayClient>(client =>
{
    client.BaseAddress = new Uri("https://payments.example.com/api/");
})
.AddResilienceHandler("payment-pipeline", pipeline =>
{
    // 1. Retry transient failures — exponential backoff + jitter avoids a retry storm
    pipeline.AddRetry(new HttpRetryStrategyOptions
    {
        MaxRetryAttempts = 3,
        BackoffType = DelayBackoffType.Exponential,
        UseJitter = true
    });

    // 2. Circuit breaker — stop hammering a clearly-failing gateway
    pipeline.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions
    {
        FailureRatio = 0.5,                        // open if ≥50% of recent calls failed
        SamplingDuration = TimeSpan.FromSeconds(30),
        MinimumThroughput = 10,                     // need enough samples before judging
        BreakDuration = TimeSpan.FromSeconds(15)    // how long the circuit stays open
    });

    // 3. Timeout — never wait indefinitely for the gateway to respond
    pipeline.AddTimeout(TimeSpan.FromSeconds(5));
});

Order matters conceptually here: a single call attempt is bounded by the timeout; if it fails, the retry strategy decides whether (and how, with backoff and jitter) to try again; and across many calls over time, the circuit breaker is watching the overall failure rate — if the gateway is clearly down, it stops even attempting new calls (retries included) until the break duration passes and it allows a test call through.

Analogy

Calling a Busy Friend

Imagine calling a friend who's briefly not answering. Calling back immediately, over and over, at the exact same interval, is annoying and doesn't help — and if a hundred people are all trying to reach that friend the same way, their phone is overwhelmed the instant it's free (thundering herd). Waiting a bit longer each time you retry, with a bit of randomness in exactly when you call back, means you're less likely to call at the exact same moment as everyone else (exponential backoff with jitter).

If you've called forty times with no answer, the sensible thing isn't to keep dialing — it's to stop for a while and try again in twenty minutes (circuit breaker: open, then a later test attempt). And you'd never let a single call ring forever without hanging up eventually (timeout) — an unanswered call that never ends is still tying up your attention the whole time.

Under the Hood

HOW A RESILIENCE PIPELINE WRAPS AN OUTGOING HTTP CALL

1. AddResilienceHandler REGISTERS A DelegatingHandler

Under the hood, the resilience pipeline is wired into HttpClient's message handler chain — the same handler-pipeline mechanism IHttpClientFactory already uses. Every request sent through this client passes through the pipeline before the actual network call happens.

▼

2. POLLY EXECUTES EACH CONFIGURED STRATEGY, IN ORDER

Microsoft.Extensions.Resilience is a thin, DI-friendly configuration layer over Polly's actual resilience strategy execution engine — timeout, retry, and circuit breaker each wrap the call in sequence, the same way middleware wraps a request in the ASP.NET Core pipeline you've already learned elsewhere in this Part.

▼

3. RETRY DELAYS USE THE FRAMEWORK'S TIMER, NOT A BLOCKING WAIT

The delay between retry attempts is implemented with an asynchronous wait, not a thread-blocking Thread.Sleep — the calling thread is free to do other work while a retry is pending, consistent with everything you already know about async/await for I/O-bound work.

Common Confusion

"More retries is always safer" — it's often the opposite

It's tempting to think "if 3 retries is good, 10 must be safer." In reality, more retries mean more total load thrown at an already-struggling dependency, and a longer total time before the caller gets a final answer (success or failure) — which can itself cascade into timeouts elsewhere in the system. A small number of retries with real exponential backoff and jitter is almost always the better trade than many retries fired in quick succession.

"A circuit breaker fixes the dependency" — it doesn't, it protects everyone else

A circuit breaker doesn't repair the failing service — it protects your app (and, indirectly, the struggling dependency) from making things worse while it's down. Opening the circuit trades "keep trying and probably keep failing slowly" for "fail fast and predictably," which is almost always better for both sides, but the underlying problem with the dependency still needs to be fixed by whoever owns it.

Common Mistakes

Mistake 1 — Fixed-delay retries with no jitter, at any meaningful scale

Delay = TimeSpan.FromSeconds(1) with no backoff and no jitter — every client retries in lockstep, which is exactly the thundering-herd setup from the hook.

Use exponential backoff (DelayBackoffType.Exponential) with UseJitter = true so retries spread out over time instead of clustering.

Mistake 2 — Retrying non-idempotent operations blindly

Automatically retrying a "charge the customer's card" POST that may have actually succeeded server-side before the response was lost — a naive retry can double-charge the customer.

Be deliberate about what's safe to retry — idempotent operations (like GETs, or POSTs designed with idempotency keys) are safe; a raw "retry everything" policy on non-idempotent writes is a real correctness risk, not just a performance one.

Mistake 3 — No timeout at all on an outbound call

Letting an HttpClient call run with an unbounded or excessively long default timeout — a single hung dependency can tie up threads and connections indefinitely, degrading your own app's responsiveness for everyone.

Always set an explicit, reasonable timeout, and treat "the dependency didn't answer in time" as its own distinct, expected failure mode to handle.

When Should I Use It?

Any outbound call to a dependency you don't fully control — a third-party API, another internal service, sometimes even a database under heavy load — deserves at least a timeout and a sensible retry policy.
Add a circuit breaker specifically when a dependency's failures tend to cluster (an outage, not just isolated blips) and you want to fail fast instead of queuing up doomed retries during that window.
Skip aggressive retry policies for calls where retrying could cause real harm (non-idempotent writes without idempotency protection) unless you've specifically designed for safe retries.
For quick internal scripts or one-off tooling with no real concurrent load, this level of resilience machinery is likely overkill — it earns its complexity in production systems under real, sustained traffic.

Mental Model

Timeout = never wait forever for one answer
Retry (backoff + jitter) = try again, but spaced out and staggered across clients
Circuit breaker = stop trying entirely for a while, once it's clearly not working

Remember: a fixed retry delay with no jitter isn't resilience — under real concurrent load, it's a mechanism for turning a small blip into a synchronized pile-on.

Key Takeaway

Microsoft.Extensions.Resilience, built on Polly, integrates directly into IHttpClientFactory via AddResilienceHandler(...) since .NET 8.
Naive fixed-delay retries from many simultaneous clients can cause a thundering herd against a recovering dependency — a synchronized wave of retries that can prolong or worsen the original outage.
Exponential backoff with jitter is the standard fix — progressively longer delays, plus randomized variation, spread retries out instead of clustering them.
A circuit breaker opens after enough consecutive/recent failures, failing fast instead of piling up more doomed requests, then periodically tests whether the dependency has recovered — this lesson covers it at an introductory level.
Timeouts bound how long you'll ever wait for a single call — never wait indefinitely for an external dependency.
A real production resilience pipeline typically combines all three — timeout, retry, circuit breaker — wrapped around the same HttpClient call.

Check Your Understanding

You've seen why naive retries can backfire and how retries, circuit breakers, and timeouts work together. Let's test your reasoning.

1. A dependency briefly fails, and 500 concurrent clients all retry after exactly the same fixed 1-second delay. What problem does this create?

A) Nothing — a fixed delay is the recommended, safest retry strategy
B) A thundering herd — all 500 clients hit the dependency again at nearly the same instant, potentially overwhelming it right as it's trying to recover
C) The retries will automatically be rejected by .NET's runtime for exceeding a hardcoded limit
D) HttpClient will throw a compile-time error if retries aren't staggered

Show answer

Correct: B

Why B is correct: This is exactly the thundering-herd scenario — identical fixed delays across many concurrent clients cause synchronized retries that can overwhelm a just-recovering dependency, potentially prolonging or worsening the original failure.

Why A is incorrect: A fixed delay with no jitter is precisely the pattern that causes this problem — it's a common mistake, not a recommended practice.

Why C is incorrect: There's no such automatic rejection mechanism — the runtime doesn't police retry timing on its own.

Why D is incorrect: This is a runtime behavioral concern under load, not something the compiler can detect.

Reinforcement: Exponential backoff with jitter exists specifically to prevent this synchronized retry spike.

2. What does "jitter" specifically add to a retry strategy that "exponential backoff" alone does not provide?

A) It guarantees the retry will always succeed on the next attempt
B) It adds a small amount of randomized variation to each client's wait time, spreading retries across a window instead of having every client with the same backoff schedule retry at the exact same moment
C) It increases the maximum number of retry attempts allowed
D) It disables the circuit breaker while retries are in progress

Show answer

Correct: B

Why B is correct: Exponential backoff alone still produces identical wait times across every client following the same schedule — they'd all still retry in a synchronized wave, just a later one. Jitter adds randomness on top, so different clients' retries land at different moments, spreading load out instead of clustering it.

Why A is incorrect: No retry strategy can guarantee success — it only improves the odds of eventually succeeding on a transient fault.

Why C is incorrect: Jitter affects timing, not the count of retry attempts — those are configured separately (e.g. MaxRetryAttempts).

Why D is incorrect: Jitter and circuit breakers are independent, complementary strategies — one doesn't disable the other.

Reinforcement: Backoff spreads retries out in time per client; jitter spreads them out across clients — both are needed to fully avoid a thundering herd.

3. After many consecutive failures calling a downstream service, a circuit breaker opens. What happens to new requests while the circuit is open?

A) They are queued indefinitely until the circuit closes again
B) They fail immediately without attempting the network call at all, until the breaker later allows a test request through to check for recovery
C) They are automatically redirected to a different, unrelated dependency
D) They are retried with an unlimited number of attempts until one succeeds

Show answer

Correct: B

Why B is correct: This is the defining behavior of an open circuit — it fails fast, deliberately skipping the actual network call, rather than piling up more slow, doomed requests against a service that's already clearly struggling. After the configured break duration, it moves to a half-open state and allows limited test traffic through.

Why A is incorrect: Open circuits fail fast, not queue — queuing would still tie up resources waiting, which defeats the purpose.

Why C is incorrect: A circuit breaker doesn't reroute traffic to alternative services on its own — that would be a separate fallback strategy, not the breaker itself.

Why D is incorrect: Unlimited retries against a dependency the circuit breaker has identified as failing is exactly the pile-up behavior circuit breakers exist to prevent.

Reinforcement: "Open" means "stop trying and fail immediately," which protects both your app and the struggling dependency.

4. Why is a timeout considered an essential, separate piece of a resilience pipeline, distinct from retries and circuit breakers?

A) It's not really necessary if retries are already configured
B) Without a timeout, a single call can hang indefinitely waiting for a response that may never come, tying up resources — a timeout bounds how long you'll ever wait for one attempt
C) Timeouts automatically fix the underlying cause of a slow dependency
D) Timeouts replace the need for a circuit breaker entirely

Show answer

Correct: B

Why B is correct: Retries and circuit breakers operate across multiple attempts or over time, but neither one bounds how long any single attempt is allowed to hang — that's specifically the timeout's job. Without it, one unresponsive call can tie up a thread/connection indefinitely, regardless of how well-configured retries or circuit breaking are.

Why A is incorrect: Retries don't help if the underlying attempt never returns at all — you need a timeout to even know an attempt has failed and should be retried.

Why C is incorrect: A timeout only bounds your own wait — it does nothing to fix whatever is making the dependency slow.

Why D is incorrect: Timeouts and circuit breakers solve different problems (bounding one call's duration vs. tracking failure patterns across many calls) and are meant to be used together, not as substitutes for each other.

Reinforcement: Timeout, retry, and circuit breaker each answer a different question about a call — none of the three alone is a complete resilience strategy.

You now understand how to build genuinely resilient calls to unreliable dependencies — retries done safely, circuit breakers, and timeouts, combined into one pipeline.

dotnetmadeeasy.com — Learn C# and .NET, the right way.