Closed, Open, Half-Open — three states standing between one struggling dependency and an outage that spreads to everything that calls it.
Advanced Part VII's resilience lesson introduced circuit breakers at an introductory level and made you a specific promise: "a fuller, deeper treatment of circuit breakers as part of broader distributed-systems resilience may appear in a later, more distributed-systems-focused part of this course." This is that lesson. You already know the shape — Closed, Open, Half-Open. What you're about to learn is exactly what happens at each state, exactly what triggers a transition between them, and precisely why this specific, humble little state machine is one of the most consequential resilience patterns in distributed systems — because without it, one struggling service can take down every service that depends on it, and every service that depends on those, rippling outward through an entire architecture.
In this lesson, you'll get the complete, precise three-state circuit breaker model — Closed, Open, and Half-Open — exactly how and why each transition happens, how this prevents cascading failures by connecting directly to the thread-pool-starvation mechanism from Advanced Part IV, and how to configure it for real using Microsoft.Extensions.Resilience/Polly, building on what you already know from Part VII.
A circuit breaker watches the calls your code makes to some dependency, and — if that dependency starts failing a lot — stops letting new calls even attempt to reach it for a while, failing them instantly instead. After a cooldown, it cautiously lets a few calls through to check whether things have improved, and decides from there whether to resume normal traffic or keep blocking it.
A circuit breaker is a stateful guard placed around calls to a dependency, implemented as a three-state finite state machine — Closed, Open, and Half-Open — named by direct analogy to an electrical circuit breaker: closed means current (requests) flows normally; open means the circuit is broken and nothing flows through; half-open is the cautious, testing state in between. The breaker tracks recent call outcomes, and its own internal state determines whether an incoming call is allowed to actually reach the dependency at all, independent of whatever retry or timeout policy might otherwise apply to that individual call.
Picture a real distributed system: Service A calls Service B, which calls Service C. Service C starts genuinely struggling — its database is overloaded, and every call to it now takes ten seconds to time out instead of failing fast. Without anything to stop it, Service B keeps sending it calls anyway (maybe with retries, making it worse), and each of those calls occupies one of Service B's threads for the full ten seconds while it waits. If Service B is fielding real traffic, its available threads fill up fast — this is precisely the thread pool starvation mechanism Advanced Part IV's thread-pool lesson covered: too many threads blocked waiting on something, too few left free to do anything else.
Now Service B itself is slow and unresponsive — not because anything is wrong with Service B's own code, but purely because it's waiting on a struggling downstream. Service A, calling Service B, experiences the exact same starvation, for the exact same reason, one hop further out. This is cascading failure: one genuinely struggling service can, with nothing to stop it, degrade every upstream caller, and every caller of those callers, rippling outward through an entire distributed system — turning one component's bad afternoon into a systemic outage.
A circuit breaker interrupts this chain at its source. Once it recognizes that a dependency is genuinely struggling — not one isolated blip, but a real pattern of failures — it stops sending it calls entirely, for a bounded period. This does two things at once: it protects the already-struggling downstream from even more load piling on while it's trying to recover, and it protects the caller from wasting threads, time, and resources on calls that are very likely doomed to fail anyway. Failing fast and predictably, instead of piling up slow, doomed calls, is exactly what breaks the chain that would otherwise cascade outward.
failures cross threshold
CLOSED ─────────────────────────▶ OPEN
▲ │
│ │ cooldown period elapses
│ test calls succeed ▼
└──────────────────────── HALF-OPEN
│
│ test calls still fail
▼
OPEN (cooldown restarts)Notice this is a genuine cycle, not a one-way trip — a circuit that opens is never stuck open forever, and a circuit that closes again is never permanently "cured." It continuously re-evaluates the dependency's real, current health.
You already saw Microsoft.Extensions.Resilience's AddCircuitBreaker at an introductory level in Part VII — here's the same configuration, now that you understand precisely what each option controls:
builder.Services.AddHttpClient<InventoryServiceClient>(client =>
{
client.BaseAddress = new Uri("https://inventory.internal/");
})
.AddResilienceHandler("inventory-breaker", pipeline =>
{
pipeline.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions
{
FailureRatio = 0.5, // Closed → Open once ≥50% of sampled calls fail
SamplingDuration = TimeSpan.FromSeconds(30), // the window the failure ratio is measured over
MinimumThroughput = 10, // need at least 10 sampled calls before judging — avoids over-reacting to a tiny sample
BreakDuration = TimeSpan.FromSeconds(15) // how long the circuit stays Open before trying Half-Open
});
});Code → Meaning → Result: FailureRatio and SamplingDuration together define exactly when Closed transitions to Open — not a raw failure count, but a failure rate within a rolling time window, with MinimumThroughput guarding against tripping the breaker off a statistically meaningless handful of calls. BreakDuration is the Open-state cooldown before the breaker automatically attempts Half-Open. The library manages the Half-Open probe's limited call volume internally — you configure the thresholds, and the state machine's transitions follow exactly the rules described above.
An order-checkout API calls a pricing service, which — under a sudden spike — starts taking 8 seconds per call instead of its usual 80 milliseconds:
| Without a circuit breaker | With a circuit breaker |
|---|---|
| Every checkout request calls the pricing service and waits up to 8 seconds for each slow response | After the failure ratio crosses threshold, the circuit opens — checkout requests fail the pricing call in milliseconds instead of waiting 8 seconds |
| Checkout's own thread pool fills up with requests all blocked waiting on the pricing service — thread pool starvation, exactly as Part IV described | Checkout's threads are never tied up waiting on a call that was never attempted — the checkout service stays responsive even while pricing struggles |
| Checkout itself becomes slow and unresponsive to its own callers (the storefront, the mobile app) — the failure has now cascaded one hop further out | Checkout can return a fast, deliberate fallback (cached prices, or a clear "pricing temporarily unavailable" response) and keep serving everything else normally |
| The pricing service, already struggling, keeps receiving full-volume traffic from every caller, making recovery harder, not easier | The pricing service gets a genuine reprieve from callers who've opened their circuits, giving it real room to recover |
This is the connective tissue between this lesson and Advanced Part IV: a circuit breaker doesn't just save a few milliseconds on doomed calls — it's specifically what prevents "one service is slow" from becoming "every service that (directly or transitively) depends on it is also slow," which is exactly how thread pool starvation propagates outward through a real distributed system.
The name isn't decorative — it's the literal mechanism the pattern is borrowed from. A home's circuit breaker (Closed) lets electricity flow normally through the wiring. If something draws a dangerous amount of current — a short circuit, an overloaded outlet — the breaker trips (Open): it physically interrupts the circuit, and no current flows through that line at all, protecting the wiring (and your house) from further damage, even though whatever caused the surge is still plugged in.
You don't just flip it back on and hope. You reset it cautiously — flip it back, and see if it holds (Half-Open). If it trips again immediately, something's still wrong, and you leave it off (back to Open) rather than repeatedly re-energizing a circuit that keeps failing. If it holds, you trust it and go back to using that line normally (Closed). The software pattern preserves every part of that behavior: protect first, test cautiously, only fully resume once you have real evidence it's safe.
This is the single most common precise misunderstanding of the state machine. Half-Open is not "back to Closed, but monitored more closely." It's "still almost as restrictive as Open, except for a handful of deliberate test calls." If you picture Half-Open as a lighter version of normal operation, you'll misjudge exactly how much traffic actually reaches the dependency during recovery testing — the honest answer is: very little, on purpose.
A retry decides "should I try this individual call again?" A circuit breaker decides "should I even let any call — retried or not — reach this dependency right now?" These operate at different scopes: retry is per-call; the circuit breaker's state persists across many calls over time. As Part VII's pipeline example already showed, they're routinely combined — retry handles an individual call's transient blip, while the circuit breaker steps in once the pattern across many calls says the dependency itself is genuinely down, at which point it stops the retries from even being attempted.
A very low FailureRatio or MinimumThroughput trips the circuit on completely ordinary, isolated transient blips — the breaker starts failing fast on a dependency that's actually fine, just having one bad moment. Tune the threshold and sampling window to reflect a genuine, sustained pattern of trouble, not a single outlier — this is exactly why MinimumThroughput exists.
Wiring up a single breaker that guards calls to two entirely different downstream services — a failure pattern in one incorrectly trips the circuit for the other, which was never actually struggling. Each distinct dependency gets its own circuit breaker instance (in Microsoft.Extensions.Resilience, this falls naturally out of attaching the pipeline per named/typed HttpClient) — health is tracked, and protection applied, per dependency.
Letting the "circuit is open" exception propagate all the way to the end user as an unhandled 500 error, identical to any other failure — this throws away the entire point of failing fast, which is the opportunity to respond quickly and deliberately instead. Catch it specifically and respond with a genuine fallback where one exists (cached data, a degraded-but-functional response) or at minimum a fast, clear "temporarily unavailable" — the breaker gives you the speed; your code still has to use it well.
Microsoft.Extensions.Resilience/Polly, already introduced in Part VII, provides a real, current, production-ready implementation via AddCircuitBreaker.You've got the full three-state model now, precisely. Let's confirm it clicked.
1. While a circuit breaker is in the Open state, what happens to a new incoming call to the guarded dependency?
Correct: B
Why B is correct: This is the defining behavior of the Open state — calls fail fast, with no attempt made to actually reach the dependency at all, protecting both the caller and the struggling dependency from more load.
Why A is incorrect: An open circuit doesn't attempt the call with adjusted settings — it skips the attempt entirely, which is the whole point of failing fast rather than failing slow.
Why C is incorrect: Queuing would still tie up resources waiting — this defeats the purpose of an open circuit, which is designed to shed load immediately, not defer it.
Why D is incorrect: The circuit breaker pattern itself doesn't automatically reroute traffic to a different service — a fallback would need to be implemented separately by the calling code, if one exists.
Reinforcement: "Open" means the dependency is never contacted at all while the circuit is in that state — that's what distinguishes it from a timeout, which still attempts the call.
2. A circuit breaker's cooldown period elapses after being Open. What happens next, precisely?
Correct: B
Why B is correct: The circuit never jumps straight from Open back to full normal operation — it always transitions to Half-Open first, testing recovery cautiously with a limited number of probe calls before deciding whether to fully close or re-open.
Why A is incorrect: This skips the entire point of Half-Open — testing before trusting. Immediately resuming full traffic risks recreating the exact pile-up that tripped the circuit in the first place if the dependency hasn't actually recovered.
Why C is incorrect: The whole design of the pattern is automatic, self-testing recovery via the cooldown-then-Half-Open cycle — it doesn't require manual intervention to reset under normal operation.
Why D is incorrect: There's no dependency on deployments in the state machine — the cooldown and subsequent Half-Open test happen automatically based on elapsed time alone.
Reinforcement: Open always transitions to Half-Open, never directly back to Closed — recovery is always tested first, never assumed.
3. In the Half-Open state, a limited test call fails. What happens?
Correct: B
Why B is correct: A failed test call in Half-Open means the dependency has not actually recovered — the circuit re-opens, and the cooldown timer restarts, exactly as though the original failure threshold had just been crossed again.
Why A is incorrect: Closing requires the test call(s) to succeed — a failed test call is precisely the signal that closing would be premature.
Why C is incorrect: The breaker doesn't give up permanently — it continues the same Open → cooldown → Half-Open cycle indefinitely, always ready to re-test.
Why D is incorrect: There's no escalating-frequency retry behavior in Half-Open — it's a single (or small, fixed number of) test call(s), and a failure sends it back to a full Open cooldown, not a faster retry loop.
Reinforcement: Half-Open's outcome is binary and clean — success closes the circuit, failure re-opens it with a fresh cooldown.
4. How does a circuit breaker connect to the thread-pool-starvation concept from Advanced Part IV?
Correct: B
Why B is correct: This is the exact cascading-failure mechanism the lesson walked through — calls to a struggling dependency tie up threads while waiting, thread pool starvation sets in, the caller itself becomes slow, and this repeats one hop further out for that caller's own callers. A circuit breaker interrupts this by failing fast instead of leaving threads blocked on doomed calls.
Why A is incorrect: They're directly connected — a struggling dependency without a circuit breaker is precisely one of the classic causes of real-world thread pool starvation.
Why C is incorrect: Circuit breakers don't touch thread pool sizing configuration at all — they prevent the starvation scenario by avoiding wasted blocking calls in the first place, not by adding more thread capacity.
Why D is incorrect: The thread pool is a completely separate CLR mechanism (Part IV) that circuit breakers don't replace — they simply reduce how much of it gets tied up waiting on doomed calls.
Reinforcement: Preventing cascading failure and preventing thread pool starvation are, in a distributed system, often the same problem described from two different angles.
5. Why is the number of test calls allowed through during Half-Open deliberately kept small, rather than immediately resuming full traffic volume?
Correct: B
Why B is correct: As the callout in How It Works explained, resuming full traffic instantly on an unproven recovery risks reproducing the original overload. A small, limited probe verifies genuine recovery before the breaker commits to letting normal volume back through.
Why A is incorrect: There's no such legal requirement — this is a deliberate engineering safety decision, not a compliance rule.
Why C is incorrect: The limited probe has a precise, well-reasoned technical purpose, as described in the lesson — it is not an arbitrary convention.
Why D is incorrect: The circuit breaker's own CPU overhead is negligible regardless of test call volume — the reason for limiting probes is about protecting the downstream dependency, not the breaker's own performance.
Reinforcement: Half-Open's restraint — testing cautiously rather than assuming recovery — is exactly what keeps the recovery check itself from becoming a second incident.
You now have the full, precise circuit breaker state machine — and understand exactly how it stops one struggling dependency from taking down an entire distributed system. Next up: distributed transactions, and why coordinating an ACID transaction across service boundaries is so much harder than it looks.
dotnetmadeeasy.com — Learn C# and .NET, the right way.