You cannot attach a debugger to production traffic across forty running instances — logs, metrics, and traces are the debugger you get instead.
Locally, when something goes wrong, you set a breakpoint, step through the code line by line, and inspect every variable along the way. That works beautifully on your machine, with one request, one thread, one process, no real users waiting. Now picture the same bug happening in production: forty running instances, thousands of concurrent requests per second, real customers mid-checkout. You cannot pause that system to single-step through it — pausing it is the outage. And yet something is failing, right now, and you need to understand why.
This is the entire reason observability exists as its own discipline: once you can't directly attach a debugger and watch, you need your system to have already been recording what it did, in a form you can query and reconstruct after the fact.
In this lesson, you'll learn the three pillars of observability — logs, metrics, and traces — what distinct question each one answers, and why distributed systems make all three genuinely necessary, not optional extras.
Observability is how well you can understand what's actually happening inside a running system, from the outside, using only the signals it produces — without needing to stop it, modify it, or guess. A system is "observable" to the degree that its external outputs (what it logs, measures, and traces) let you reconstruct and explain its internal behavior after something happens.
Observability is conventionally built from three complementary kinds of telemetry, each answering a different question about your running system:
ILogger<T> and structured loggingLogs alone tell you a lot about one specific event, but they're terrible at answering aggregate questions at scale — "what's my p99 latency across the last hour?" is not something you want to answer by eyeballing millions of individual log lines; that's exactly the kind of question a single log line can't efficiently answer. Metrics alone tell you that something is wrong ("error rate just spiked to 12%") but not why — they compress away the specific detail you'd need to actually diagnose the cause. And in a distributed system, neither logs nor metrics from any single service show you the whole story of one user-facing request that hopped across five different services — each service's own logs only show its own piece, in isolation, with no obvious way to stitch the pieces back together into one coherent narrative.
Logs, metrics, and traces aren't competing options — they're complementary, each covering the others' blind spots. A metric spike tells you something's wrong and roughly when; a trace shows you exactly which downstream call in one affected request was actually slow or failing; logs at that point in that trace give you the precise detail — the exception message, the specific input — needed to actually fix it. Together, they let you go from "something is wrong" to "here's exactly what, where, and why" without ever needing to reproduce the problem locally.
A user's checkout request hits your system:
API Gateway → OrderService → PaymentService → InventoryService
│ │ │ │
LOGS LOGS LOGS LOGS (each service's own event record)
│ │ │ │
└──────────────┴──────────────┴────────────────┘
│
ONE TRACE
(the whole journey, stitched together,
showing exactly where the 4-second delay was)
Meanwhile, across ALL requests over the last hour:
METRICS
(p99 latency, error rate, requests/sec —
"is this normal, or is something degrading?")A trace answers "what happened on this one journey." Metrics answer "what's the overall pattern across everything." Logs answer "what exactly happened at this one point." None of the three substitutes for the others.
You already know this pillar in depth: ILogger<T> and structured logging with named message templates, from earlier in this course. Every log entry is a discrete record — "at this timestamp, this event happened, with these structured properties attached." That's exactly the raw material observability's log pillar is built from; nothing new to learn here mechanically, just a new frame for why it matters at this scale.
A metric is a number, tracked over time, usually aggregated: a counter (total requests handled), a gauge (current queue depth right now), or a histogram (the distribution of request latencies, letting you ask for the p50, p95, p99 — "95% of requests finished faster than X"). Metrics are deliberately compact — instead of storing every individual event, they store aggregated statistics, which is exactly why they scale to answering "how is the whole system doing right now" without drowning in raw event volume the way scanning every log line would.
A trace represents one logical operation — typically one incoming request — as it flows through your system, made up of individual spans: one span for the time spent in the API gateway, another for the call to OrderService, another nested inside it for the call to PaymentService, and so on. Every span carries a shared identifier tying it back to the same overall trace, so a tracing tool can reconstruct the entire journey — in order, with exact timing — even though the work happened across several completely separate processes.
The same slow checkout, seen through each of the three pillars:
METRIC (aggregated, across all checkouts, last 15 minutes)
checkout.duration.p99 = 4200ms ← up from a usual ~800ms. Something is wrong, system-wide.
TRACE (one specific slow checkout request)
├─ API Gateway 12ms
├─ OrderService.CreateOrder 45ms
│ └─ PaymentService.Charge 4010ms ← here. This span is where the time went.
└─ InventoryService.Reserve 38ms
LOG (from inside PaymentService, at that exact span)
warn: PaymentService[0]
Payment gateway retry 3/3 for OrderId 88213 — upstream timeoutThe metric told you that something changed. The trace told you exactly where in the request the time went — one specific downstream call, not the whole pipeline. The log, scoped to that exact span, told you why — the payment gateway itself was timing out and being retried. No single pillar gets you from "something's wrong" to "here's the root cause" alone.
Consider an on-call engineer paged at 2am because an alert fired: error rate on the checkout endpoint crossed 5% (a metric threshold). Without traces, they'd have to guess which of a dozen downstream services is involved, and start manually grepping through each service's own logs hoping to spot a pattern — slow, and error-prone under time pressure. With tracing wired in, they instead open a handful of the actual failed traces from that time window, immediately see that every one of them failed inside the same downstream InventoryService call, follow that span directly to the exact log lines emitted at that failure point, and see the actual exception: a specific product ID causing a null-reference deep in inventory lookup logic. What could have been an hour of guessing becomes a few minutes of following a direct thread from symptom to cause — this is the entire practical payoff of having all three pillars in place before you need them, not after.
Metrics are like a hospital's dashboard of vital signs across the whole building right now — average wait time, beds occupied, admissions per hour. It tells staff "something's off in the ER tonight" without describing any one patient.
Traces are like one specific patient's full visit record — check-in, triage, which doctor saw them, which department they were sent to, how long each step took, in order. It reconstructs one patient's entire journey through the building.
Logs are like the individual notes a nurse or doctor writes at each specific step — "10:42pm, patient reports chest pain, ordered ECG." Detailed, timestamped, but only about one moment.
You'd never run a hospital on just one of these. The vitals dashboard tells you to look closer; the patient's visit record tells you where in their visit to look; the doctor's notes tell you exactly what was found there.
Monitoring usually means watching a known set of predefined signals for known failure modes — "alert me if CPU exceeds 90%." Observability is broader: it's about having enough recorded detail to answer questions you didn't think to ask in advance, when a genuinely new, unanticipated kind of problem shows up. Good monitoring tells you something's wrong; genuine observability lets you actually figure out why, even for a failure mode nobody explicitly built a dashboard for.
Heavy, detailed logging feels like it should be enough — and for a single-process app, it often mostly is. But logging alone doesn't give you cheap aggregate answers at scale (that's what metrics are for), and it doesn't natively stitch together one request's path across multiple separate services (that's what traces are for). Once a single user-facing request can touch several different processes, log volume alone stops being a substitute for the other two pillars.
Trying to reconstruct a multi-service request's journey by manually correlating timestamps across several services' separate log streams — slow, error-prone, and it doesn't scale past a handful of services.
Add tracing so the request's journey is captured directly, with an explicit shared identifier tying every service's contribution to the same one trace.
Treating logs, metrics, and traces as something to bolt on retroactively, once a production issue has already been painful to diagnose without them.
Wire up all three pillars as a normal part of building the system, before you need them — you can't retroactively generate a trace for a request that already finished and wasn't being traced at the time.
Seeing "error rate spiked" and stopping there, without following it down into a trace and the relevant logs to find the actual root cause.
Use metrics to know that and roughly when something's wrong, then use traces and logs to find out why — each pillar plays its own role in that chain.
ILogger<T> and structured logging.You've seen why production systems demand a different kind of debugging. Let's confirm you understand how the three pillars divide the work.
1. An on-call engineer wants to know: "across the last hour, what's the 99th-percentile response time for the checkout endpoint?" Which pillar is the right tool for this specific question?
Correct: B
Why B is correct: This is exactly the kind of "how much/how fast, in aggregate" question metrics are built for — a latency histogram is already computing percentiles across many events, making this a cheap, direct lookup rather than a scan through raw data.
Why A is incorrect: Logs could theoretically be used to compute this, but doing so at scale is exactly the inefficient approach metrics exist to avoid.
Why C is incorrect: A single trace shows one request's journey, not an aggregate statistic across an hour of many requests.
Why D is incorrect: Metrics are specifically designed to answer aggregate questions like this efficiently.
Reinforcement: Reach for metrics whenever the question is about volume, rate, or distribution across many events, not any single one.
2. A single user-facing request passes through an API gateway, an order service, and a payment service before returning. One specific request was unusually slow. Which pillar is specifically designed to show exactly which of those three hops the delay occurred in?
Correct: C
Why C is correct: This is exactly a trace's job — reconstructing one specific request's path across multiple services, with per-span timing, using a shared identifier that ties each service's contribution back to the same overall journey.
Why A is incorrect: This is technically possible but exactly the slow, error-prone, non-scaling approach the lesson describes as the reason tracing exists in the first place.
Why B is incorrect: Metrics show aggregate trends across many requests, not the specific path of one particular request.
Why D is incorrect: Reproducing a distributed, timing-dependent issue locally is often impractical or impossible — this is precisely why observability tooling exists as an alternative to live debugging.
Reinforcement: Traces are the pillar purpose-built for "where, specifically, in this one request's journey did the problem happen."
3. Why does observability become more important, not less, as a system becomes more distributed across many services and instances?
Correct: B
Why B is correct: This is the lesson's central point — a distributed, live production system can't be paused and inspected the way local code can be stepped through, and a single request may span multiple processes with no one place to directly "watch" it happen. Logs, metrics, and traces become the tools that fill that gap.
Why A is incorrect: Distributed systems are generally harder, not easier, to debug with a traditional live debugger — there's no single process to attach to that captures the whole picture.
Why C is incorrect: Observability matters for single-instance systems too, but its necessity grows sharply once requests span multiple services — the opposite of this option's claim.
Why D is incorrect: More services generally means more potential failure points and more places a request can go wrong, not fewer — this is part of why distributed systems need stronger observability, not weaker.
Reinforcement: The less directly you can observe a system live, the more its own recorded telemetry has to do the work instead.
4. Which statement correctly distinguishes monitoring from the broader idea of observability, as covered in this lesson?
Correct: B
Why B is correct: Monitoring is generally about watching for known, predefined conditions ("alert if X exceeds Y"). Observability is broader — having enough recorded, queryable detail to diagnose problems you didn't specifically anticipate or build a dashboard for in advance.
Why A is incorrect: They're related but meaningfully distinct, as described above — treating them as identical misses the actual point of the distinction.
Why C is incorrect: Both monitoring and observability can draw on logs, metrics, and traces — the distinction is about known-vs-unanticipated questions, not which pillar each uses.
Why D is incorrect: The distinction, while sometimes overused in marketing contexts, reflects a genuine, real difference in what each practice lets you do.
Reinforcement: Good monitoring catches known problems; genuine observability lets you diagnose the ones nobody thought to specifically watch for.
You now understand the three pillars of observability and why distributed systems make them essential, not optional. Next: OpenTelemetry, the modern, standard way to implement all three together.
dotnetmadeeasy.com — Learn C# and .NET, the right way.