Testing tries to catch a failure before it happens. Production debugging figures out what happened once it did anyway. This Part has now taught you both halves — this lesson is where they meet.
Fourteen lessons ago, Part XI opened with unit tests (308) — the smallest, cheapest way to catch a bug before it ships. It built up through integration testing, mocking, and testcontainers (309-311), through API and performance testing (312-313), to load testing (314), which put a whole system under realistic pressure on purpose, to watch it bend before a real customer ever could. Then lesson 315 turned the page entirely: no more controlled rehearsals. Real production, real failures, no debugger. Lessons 316 through 320 handed you five specific, sharp tools — one for a leak, one for pathological CPU, one for a genuine deadlock, one for thread pool starvation, one for a database bottleneck — each with its own signature, its own diagnostic command, and its own fix.
None of that is worth much until you can do it under pressure, in the right order, on a system you're seeing fail for the first time. This capstone lesson walks one complete, realistic incident from first alert to root cause to fix — using lesson 315's triage framework to narrow the search, systematically ruling out each of lessons 316-320's failure modes with real evidence instead of guesswork, and closing the loop by asking what this Part's first half, the testing discipline of lessons 308-314, should have caught before any of it ever reached a real customer.
Incident analysis is the disciplined, end-to-end process of taking a live production problem from "something's wrong" to "here's exactly why, here's the fix, and here's what should have caught it earlier" — using evidence at every step, never a hunch dressed up as a conclusion.
A rigorous incident analysis applies lesson 315's triage framework first to narrow the space of plausible causes, then systematically tests each candidate failure mode from lessons 316-320 against real diagnostic evidence — ruling each one in or out explicitly — until exactly one root cause is confirmed, not merely suspected. It closes not with a fix alone, but with a look backward at lessons 308-314's testing practices: which of them, applied earlier, would have caught this specific class of problem before it ever reached production.
Here is the uncomfortable truth this whole Part has been building toward: from a customer's point of view, "the app is timing out" is the entire observable symptom for a memory leak finally exhausting available memory (316), pathological CPU burning every core (317), a genuine deadlock (318), thread pool starvation (319), and a database bottleneck (320) alike. Guessing which one it is, and reaching for that lesson's fix on a hunch, is a coin flip with five sides. Worse, several of these causes actively chain into each other — lesson 320 showed a slow query triggering lesson 319's starvation; a slow-building leak (316) can eventually starve available memory for everything else running in the same process. Ad hoc firefighting doesn't scale against that kind of overlap.
The fix isn't a smarter guess — it's a systematic elimination process, using the specific, distinguishing evidence each of lessons 316-320 already taught you to look for. Every failure mode in this Part has at least one piece of evidence that confirms or rules it out cleanly: a growing gcdump diff, a flame graph's dominant method, a confirmed syncblk cycle, a ThreadPool queue length, a captured SQL statement's execution plan. Walk through them in a sensible order, and the coin flip disappears.
Here is the distinguishing signal for each failure mode this Part covered, side by side — this is the map an incident should be walked against, not five separate lessons in isolation:
Notice the shape of this table: it's the same three-question triage framework from lesson 315 — what changed, what the graphs show, which dependency is implicated — refined one level further into "and once you know roughly which category, here's the one piece of evidence that actually confirms it." That's what makes this a decision tree rather than five unrelated checklists.
A checkout API — the same shape of system this whole Advanced tier has been building toward — starts timing out under what looks like completely ordinary Tuesday-afternoon traffic. No unusual spike. Here's the investigation, in the order it actually happened:
dotnet-counters shows a stable, unremarkable heap size — no upward trend at all. Ruled out. A leak takes hours or days to matter; this started climbing within minutes of the deploy, an inconsistent timeline.dotnet-counters confirms it: ThreadPool Queue Length is climbing, Thread Count growing only slowly. Starvation is real and measurable — but lesson 319 also taught that starvation is very often a downstream symptom of something else blocking a thread. The question becomes: blocking on what?.Include() that used to eager-load a customer's saved payment methods. What used to be one query is now one query per payment method — an N+1 pattern, invisible in code review, catastrophic under real order volume. Each of those extra round trips holds a pooled connection open a little longer, and the checkout handler blocks synchronously waiting on the whole thing, tying up a thread-pool thread for the duration — exactly lesson 319's classic trigger, with lesson 320's N+1 pattern as its true upstream cause..Include() in a routine deploy caused an N+1 database bottleneck (320), which — because the code path blocked synchronously waiting on it — starved the ThreadPool (319) of capacity for every other checkout in flight. Not a leak, not pathological CPU, not a deadlock. Ruled out systematically, not guessed.
// The regression, introduced by the deploy — a missing .Include()
// silently turned one query into an N+1 pattern (lesson 320):
var order = await db.Orders
// .Include(o => o.Customer.PaymentMethods) ← accidentally removed
.FirstOrDefaultAsync(o => o.Id == orderId);
// The fix — restore eager loading, and stop blocking the request
// thread synchronously on the checkout confirmation step that made
// the starvation (lesson 319) so much worse than the query alone:
var order = await db.Orders
.Include(o => o.Customer.PaymentMethods)
.FirstOrDefaultAsync(o => o.Id == orderId);
var confirmation = await paymentGateway.ConfirmAsync(order); // was .ResultMeaning: Neither half of the fix alone was the whole story. Restoring .Include() removes the N+1 pattern at its root; removing the blocking .Result call means that even a slower-than-ideal database call in the future won't tie up a thread-pool thread the same way again. Before shipping either change, the team re-ran the same performance test suite from lesson 313 against a copy of production-scale order data — exactly the check that should have caught this the first time.
This is the question the incident isn't complete without asking. The N+1 pattern here was a one-line diff — removing an .Include() during an unrelated refactor — the kind of change that sails through a unit test (308) with a mocked repository (310) that was never going to notice an extra round trip, because there was no real database underneath it to measure against. Integration testing against a real, ephemeral database via testcontainers (309, 311) would have exercised the actual query, but with only a handful of seeded rows in a test fixture, the difference between one query and a dozen is invisible in wall-clock time — passes fine, ships fine. Performance testing (313), run specifically against a dataset shaped like production — customers with realistic numbers of saved payment methods, not three neat test records — is exactly the check built to catch a query whose cost scales badly with data shape, and load testing (314), replaying realistic concurrent checkout volume, is exactly what would have shown the thread pool starvation pattern forming under pressure, before a single real customer ever felt it. The gap wasn't a missing test category. It was a performance/load test that either didn't exist for this endpoint, or ran against unrealistic data — the specific blind spot lessons 313 and 314 exist to close.
A doctor presented with "the patient has a fever" doesn't reach for the first plausible-sounding illness and start treatment. They run a structured process called differential diagnosis: list every condition that could produce this symptom, then use specific tests — a blood panel, an imaging scan, a targeted question — to rule each one out or in, one at a time, until exactly one explanation survives every test that could have contradicted it. "Feels like an infection" isn't a diagnosis; a confirmed pathogen, isolated by a specific test that couldn't be explained any other way, is. This Part's five failure modes are exactly this list of candidate conditions, and lessons 316-320 each handed you the specific test that rules one in or out — not a symptom checklist to eyeball, but real evidence to run.
Each lesson in Part XI is, underneath, a specific answer to a question this incident-analysis discipline needs answered — worth laying out as one connected picture, the same way lesson 295 closed out Part IX:
| Question a real incident raises | This Part's answer |
|---|---|
| How do we catch a bug before it ever reaches production, cheaply? | Unit testing, mocking (308, 310) |
| How do we test against something that behaves like a real dependency? | Integration testing, testcontainers (309, 311) |
| How do we verify a whole API contract, not just one method? | API testing (312) |
| How do we know a change is fast enough before shipping it? | Performance testing (313) |
| How do we know the system survives realistic concurrent traffic? | Load testing (314) |
| Once it's live, how do we investigate without a debugger? | Non-invasive diagnostics, triage framework (315) |
| What if memory is quietly climbing? | Managed leak diagnosis via dotnet-gcdump (316) |
| What if CPU is pinned? | Legitimate vs. pathological CPU via dotnet-trace (317) |
| What if specific requests are permanently stuck? | Deadlock confirmation via dotnet-dump/syncblk (318) |
| What if everything is uniformly slow but still moving? | ThreadPool starvation via dotnet-counters (319) |
| What if only database-touching endpoints are slow? | Query/pool diagnosis via EF Core logging (320) |
| How does all of this fit together in one real incident? | This lesson |
In the walkthrough above, thread pool starvation was real and measurable at step 5 — and it would have been tempting to stop there, ship a SetMinThreads mitigation, and call the incident closed. Lesson 319 explicitly warned against exactly this: starvation is very often a symptom of something else blocking a thread, not the root cause itself. Stopping at the first confirmed-but-not-necessarily-final piece of evidence is how the same incident recurs a week later.
This walkthrough involved two of this Part's five failure modes at once, one causing the other. Ruling out the other three (leak, high CPU, deadlock) cleanly, and then correctly linking starvation to its actual upstream cause rather than treating it as the final answer, is exactly the systematic elimination this lesson has been building toward — not a search for a single tidy label.
Opening dotnet-trace immediately on instinct, without first checking what changed, what the graphs show, and which endpoint is implicated — burning time on a tool that may not even apply to the actual failure mode.
Triage first, exactly as lesson 315 taught — it's what narrowed this incident from five candidate failure modes down to two worth real investigation, in minutes rather than hours.
Confirming thread pool starvation via dotnet-counters and treating that confirmation as the finished diagnosis, shipping only a SetMinThreads mitigation.
Ask what's blocking the thread in the first place — starvation is frequently a symptom of an upstream cause (a slow query, a database bottleneck) that needs its own fix, not just a higher thread floor.
Shipping the fix, closing the ticket, and moving on — leaving the exact same class of bug (a silently dropped eager-load, invisible without realistic data volume) free to recur on the next refactor.
Close every incident by asking which of lessons 308-314's testing practices — run against realistic data and realistic concurrency — would have caught this before it shipped, and make sure that gap actually gets closed.
You've walked one complete incident from first alert to confirmed root cause, using every tool this Part built. Let's confirm it all ties together — and close out Part XI.
1. In this lesson's walkthrough, dotnet-counters confirms thread pool starvation is genuinely occurring. Why doesn't the investigation stop there?
Correct: B
Why B is correct: This is exactly the lesson's central point about chained failure modes — confirming starvation is real evidence, but lesson 319 already taught that starvation is frequently a symptom of an upstream blocking cause, which here turned out to be an N+1 database bottleneck (320).
Why A is incorrect: This is precisely Mistake 2 the lesson warns against — stopping at the first confirmed symptom instead of chasing it to its actual root cause.
Why C is incorrect: dotnet-counters reliably reports ThreadPool queue length and thread count — the issue isn't reliability of the tool, it's that a confirmed symptom still needs its upstream cause identified.
Why D is incorrect: The walkthrough shows exactly the opposite — the two failure modes are directly connected, with the database bottleneck causing the starvation, not excluding it.
Reinforcement: A confirmed symptom is not automatically the root cause — always ask what's causing it.
2. In the walkthrough, how was a genuine deadlock (218/318) ruled out without even needing a full dotnet-dump analysis?
Correct: B
Why B is correct: This applies lesson 318's key distinguishing test directly — a genuine deadlock never recovers on its own, so observing the same requests eventually complete (even slowly) is sufficient evidence to rule out a true deadlock without needing a full dump analysis.
Why A is incorrect: Deadlocks absolutely can occur in ASP.NET Core (the two-lock ordering deadlock from lesson 218 has nothing to do with SynchronizationContext) — this isn't the reasoning used, and it isn't accurate.
Why C is incorrect: While a genuine deadlock does show near-idle CPU for the affected threads, that's not what ruled it out in the walkthrough — the recovery-over-time observation was the specific test applied.
Why D is incorrect: The walkthrough describes a gradual p99 climb with flat error rate, not a hard step change — and this graph shape was used earlier, for triage, not as the specific test that ruled out a deadlock.
Reinforcement: "Does it ever recover?" is the fast, cheap first test for a deadlock — no dump required to at least tentatively rule it out.
3. According to the Real-World Example, why did the N+1 regression pass unit tests (308) and even basic integration tests (309, 311) without being caught?
Correct: B
Why B is correct: This is exactly the gap the Real-World Example identifies — mocked or lightly-seeded tests can't surface a query-count problem that only matters at realistic data volume, which is precisely why performance testing (313) and load testing (314), run against production-shaped data and concurrency, are the specific checks that would have caught it.
Why A is incorrect: The lesson doesn't claim these test types are universally incapable — it identifies a specific blind spot (unrealistic data volume/mocking) that a different test type (313/314) is built to cover.
Why C is incorrect: The example explicitly describes the change passing through unit and integration testing — implying tests existed and ran, just without catching this specific class of issue.
Why D is incorrect: The regression was introduced during a routine refactor before the deploy — well within the window where testing, if it exercised realistic data shape, could have caught it.
Reinforcement: Different test types catch different classes of bug — a data-volume-sensitive regression specifically needs a test run against realistic data volume to surface.
4. A team fixes a production incident, ships the fix, and closes the ticket without further discussion. What does this lesson say is missing from that response?
Correct: B
Why B is correct: This is exactly Mistake 3 and the lesson's closing synthesis — an incident isn't fully closed by a fix alone; asking which testing practice from lessons 308-314 should have caught it, and closing that gap, is what actually prevents recurrence.
Why A is incorrect: The lesson explicitly frames shipping the fix without this follow-up question as an incomplete response, leaving the same class of bug free to recur.
Why C is incorrect: Restarting the service is unrelated to closing the loop on testing coverage — it's an operational action, not the reflective step the lesson is asking for here.
Why D is incorrect: SetMinThreads is explicitly framed elsewhere in this Part (lesson 319) as a blunt mitigation, not a permanent fix — and it addresses only the starvation symptom, not the underlying testing gap or the database-bottleneck root cause.
Reinforcement: Closing an incident well means asking what testing gap allowed it, not just shipping the fix and moving on.
5. How does this lesson characterize the relationship between testing (lessons 308-314) and production debugging (lessons 315-320)?
Correct: B
Why B is correct: This is the lesson's explicit closing synthesis — testing and production debugging are framed as complementary, equally necessary expressions of the same underlying discipline: careful, evidence-based reasoning, applied before a change ships and again after a failure occurs.
Why A is incorrect: The lesson's own incident walkthrough demonstrates the opposite — even with testing in place, a real production failure still occurred and still required production debugging to resolve.
Why C is incorrect: The lesson explicitly avoids ranking one as more important than the other — it frames them as equally necessary, complementary halves of the same discipline, not a hierarchy.
Why D is incorrect: The lesson goes out of its way to connect the two deliberately — the whole capstone is built around showing they're the same discipline applied at different points in a system's life, not an arbitrary pairing.
Reinforcement: Verification before shipping, diagnosis after failure — two applications of the same careful, evidence-based engineering mindset.
That closes Part XI — Testing & Prod Eng. You now understand not just how to write a good test and not just how to diagnose a live production failure, but why both exist together: methodical verification before a change ships, and methodical diagnosis when something gets through anyway, are the same engineering discipline, applied at two different moments in a system's life.
dotnetmadeeasy.com — Learn C# and .NET, the right way.