Resilience in .NET: Retries, Timeouts and Circuit Breakers

A dependency that fails once is a blip. A thousand clients retrying it at the exact same instant is an outage you caused.

Your API calls a third-party shipping-rate service on every checkout. One afternoon, that service hiccups for four seconds under its own load spike — a completely normal, survivable blip for a well-behaved client to just retry past. Except every one of your app's in-flight requests notices the failure at roughly the same moment, and every one of them retries after exactly the same fixed one-second delay you hardcoded. A wall of retries lands on the shipping service at the exact same instant, right as it's trying to recover — and that wave of simultaneous retries is often enough to knock it back down, or keep it down longer than the original blip ever would have. You didn't just fail to handle a transient fault. You made it worse.

In this lesson, you'll learn the three pillars of resilient calls to external dependencies — retries (done correctly, with exponential backoff and jitter), circuit breakers, and timeouts — and how to wire all three into IHttpClientFactory using Microsoft.Extensions.Resilience.

What Is It?

The Simple Explanation

Resilience is the practice of assuming, up front, that the external things your code depends on — other APIs, databases, third-party services — will occasionally fail, slow down, or become temporarily unreachable, and writing your code so that when that happens, it degrades gracefully instead of falling over or making the problem worse.

The Technical Definition

Modern .NET builds resilience on top of Polly, the long-established .NET resilience and transient-fault-handling library, via the Microsoft.Extensions.Resilience package. Since .NET 8, this integrates directly into IHttpClientFactory — the typed/named client infrastructure you already know from the Intermediate course — through AddResilienceHandler(...), letting you attach a named pipeline of resilience strategies (retry, circuit breaker, timeout, and others) to any HttpClient registration with a few lines of configuration.

Why Does It Exist?

The Problem — Networks and Dependencies Fail, Constantly, Briefly

Once your app makes real network calls — to a database, a third-party API, another internal service — it inherits every way a network call can go wrong: a momentary DNS hiccup, a server briefly overloaded, a load balancer mid-deployment, a packet dropped somewhere in between. Most of these are transient — gone in milliseconds to seconds, not permanent outages. Code that treats every failure as fatal (bubble the exception straight up, fail the whole request) throws away perfectly recoverable situations. Code that retries naively — same delay, every time, for every client — can turn one dependency's bad five seconds into everyone's bad five minutes, through the exact thundering-herd mechanism in the hook above.

The Solution — Assume Failure, Handle It Deliberately

Resilience isn't one trick — it's a small set of complementary strategies, each solving a different failure shape: retries recover from brief blips, circuit breakers stop hammering something that's clearly down, and timeouts stop you from waiting forever on something that will never answer. Used together, they let your app survive the kind of failures that are completely normal in any system with real network dependencies.

Big Picture

Your code
    ↓
Timeout        — "never wait forever for an answer"
    ↓
Retry (with backoff + jitter) — "a brief blip? try again, spaced out sensibly"
    ↓
Circuit Breaker — "clearly and repeatedly failing? stop hammering it, fail fast for a while"
    ↓
The external dependency (API, database, third-party service)

These three wrap around a single outbound call, each addressing a different question: how long do I wait (timeout), do I try again on failure and how (retry), and should I even bother trying right now given recent history (circuit breaker)?

How It Works

Retries — And the Thundering Herd Problem

Circuit Breakers — Failing Fast Instead of Piling Up

Timeouts — Never Wait Forever

A call with no timeout can, in the worst case, hang indefinitely — tying up a thread, a connection, and the caller's own responsiveness while waiting for an answer that may never come. A timeout guarantees the call gives up after a bounded, known amount of time, converting an indefinite hang into a definite, handleable failure.

Simple Example

A single retry strategy, attached to a named HttpClient via AddResilienceHandler:

builder.Services.AddHttpClient("ShippingApi", client =>
{
    client.BaseAddress = new Uri("https://shipping.example.com/");
})
.AddResilienceHandler("shipping-retry", pipeline =>
{
    pipeline.AddRetry(new HttpRetryStrategyOptions
    {
        MaxRetryAttempts = 3,
        BackoffType = DelayBackoffType.Exponential,
        UseJitter = true,                         // spreads retries out — avoids thundering herd
        Delay = TimeSpan.FromSeconds(1)            // base delay before exponential growth + jitter
    });
});

BackoffType = DelayBackoffType.Exponential makes each retry wait longer than the last; UseJitter = true adds the randomized variation that spreads many clients' retries across a window instead of one synchronized spike — this single flag is the direct fix for the thundering-herd scenario from the hook.

Real-World Example

A payment-gateway typed client, combining retry, circuit breaker, and timeout into one resilience pipeline — the standard shape for a real production integration with an external dependency:

public class PaymentGatewayClient(HttpClient httpClient)
{
    public async Task<bool> ChargeAsync(string cardToken, decimal amount, CancellationToken ct)
    {
        var json = $$"""{"token": "{{cardToken}}", "amount": {{amount}}}""";
        var content = new StringContent(json, System.Text.Encoding.UTF8, "application/json");

        var response = await httpClient.PostAsync("charges", content, ct);
        return response.IsSuccessStatusCode;
    }
}

builder.Services.AddHttpClient<PaymentGatewayClient>(client =>
{
    client.BaseAddress = new Uri("https://payments.example.com/api/");
})
.AddResilienceHandler("payment-pipeline", pipeline =>
{
    // 1. Retry transient failures — exponential backoff + jitter avoids a retry storm
    pipeline.AddRetry(new HttpRetryStrategyOptions
    {
        MaxRetryAttempts = 3,
        BackoffType = DelayBackoffType.Exponential,
        UseJitter = true
    });

    // 2. Circuit breaker — stop hammering a clearly-failing gateway
    pipeline.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions
    {
        FailureRatio = 0.5,                        // open if ≥50% of recent calls failed
        SamplingDuration = TimeSpan.FromSeconds(30),
        MinimumThroughput = 10,                     // need enough samples before judging
        BreakDuration = TimeSpan.FromSeconds(15)    // how long the circuit stays open
    });

    // 3. Timeout — never wait indefinitely for the gateway to respond
    pipeline.AddTimeout(TimeSpan.FromSeconds(5));
});

Order matters conceptually here: a single call attempt is bounded by the timeout; if it fails, the retry strategy decides whether (and how, with backoff and jitter) to try again; and across many calls over time, the circuit breaker is watching the overall failure rate — if the gateway is clearly down, it stops even attempting new calls (retries included) until the break duration passes and it allows a test call through.

Analogy

Under the Hood

Common Confusion

"More retries is always safer" — it's often the opposite

It's tempting to think "if 3 retries is good, 10 must be safer." In reality, more retries mean more total load thrown at an already-struggling dependency, and a longer total time before the caller gets a final answer (success or failure) — which can itself cascade into timeouts elsewhere in the system. A small number of retries with real exponential backoff and jitter is almost always the better trade than many retries fired in quick succession.

"A circuit breaker fixes the dependency" — it doesn't, it protects everyone else

A circuit breaker doesn't repair the failing service — it protects your app (and, indirectly, the struggling dependency) from making things worse while it's down. Opening the circuit trades "keep trying and probably keep failing slowly" for "fail fast and predictably," which is almost always better for both sides, but the underlying problem with the dependency still needs to be fixed by whoever owns it.

Common Mistakes

Mistake 1 — Fixed-delay retries with no jitter, at any meaningful scale

Delay = TimeSpan.FromSeconds(1) with no backoff and no jitter — every client retries in lockstep, which is exactly the thundering-herd setup from the hook.

Use exponential backoff (DelayBackoffType.Exponential) with UseJitter = true so retries spread out over time instead of clustering.

Mistake 2 — Retrying non-idempotent operations blindly

Automatically retrying a "charge the customer's card" POST that may have actually succeeded server-side before the response was lost — a naive retry can double-charge the customer.

Be deliberate about what's safe to retry — idempotent operations (like GETs, or POSTs designed with idempotency keys) are safe; a raw "retry everything" policy on non-idempotent writes is a real correctness risk, not just a performance one.

Mistake 3 — No timeout at all on an outbound call

Letting an HttpClient call run with an unbounded or excessively long default timeout — a single hung dependency can tie up threads and connections indefinitely, degrading your own app's responsiveness for everyone.

Always set an explicit, reasonable timeout, and treat "the dependency didn't answer in time" as its own distinct, expected failure mode to handle.

When Should I Use It?

• Any outbound call to a dependency you don't fully control — a third-party API, another internal service, sometimes even a database under heavy load — deserves at least a timeout and a sensible retry policy.
• Add a circuit breaker specifically when a dependency's failures tend to cluster (an outage, not just isolated blips) and you want to fail fast instead of queuing up doomed retries during that window.
• Skip aggressive retry policies for calls where retrying could cause real harm (non-idempotent writes without idempotency protection) unless you've specifically designed for safe retries.
• For quick internal scripts or one-off tooling with no real concurrent load, this level of resilience machinery is likely overkill — it earns its complexity in production systems under real, sustained traffic.

Mental Model

Key Takeaway

Check Your Understanding

You've seen why naive retries can backfire and how retries, circuit breakers, and timeouts work together. Let's test your reasoning.

You now understand how to build genuinely resilient calls to unreliable dependencies — retries done safely, circuit breakers, and timeouts, combined into one pipeline.