Observability in .NET: Logs, Metrics and Traces Explained

AdvancedLesson 244 of 315·Production .NET·20 min read·Updated 17 Oct 2026

You cannot attach a debugger to production traffic across forty running instances — logs, metrics, and traces are the debugger you get instead.

Locally, when something goes wrong, you set a breakpoint, step through the code line by line, and inspect every variable along the way. That works beautifully on your machine, with one request, one thread, one process, no real users waiting. Now picture the same bug happening in production: forty running instances, thousands of concurrent requests per second, real customers mid-checkout. You cannot pause that system to single-step through it — pausing it is the outage. And yet something is failing, right now, and you need to understand why.

This is the entire reason observability exists as its own discipline: once you can't directly attach a debugger and watch, you need your system to have already been recording what it did, in a form you can query and reconstruct after the fact.

In this lesson, you'll learn the three pillars of observability — logs, metrics, and traces — what distinct question each one answers, and why distributed systems make all three genuinely necessary, not optional extras.

What Is It?

The Simple Explanation

Observability is how well you can understand what's actually happening inside a running system, from the outside, using only the signals it produces — without needing to stop it, modify it, or guess. A system is "observable" to the degree that its external outputs (what it logs, measures, and traces) let you reconstruct and explain its internal behavior after something happens.

The Technical Definition — The Three Pillars

Observability is conventionally built from three complementary kinds of telemetry, each answering a different question about your running system:

Logs

Discrete, timestamped records of individual events
"What happened, and when, in this specific instance?"
You already know this — ILogger<T> and structured logging

Metrics

Numeric measurements aggregated over time
"How much? How many? How fast, on average, right now?"
Request counts, latency percentiles, error rates, queue depth

Traces

The journey of one specific request across components/services
"What actually happened, in order, across everything this one request touched?"
Especially critical once a request crosses multiple services

Why Does It Exist?

The Problem — Each Pillar Alone Has a Real Blind Spot

Logs alone tell you a lot about one specific event, but they're poorly suited to answering aggregate questions at scale — "what's my p99 latency across the last hour?" is not something you want to answer by eyeballing millions of individual log lines; that's exactly the kind of question a single log line can't efficiently answer. Metrics alone tell you that something is wrong ("error rate just spiked to 12%") but not why — they compress away the specific detail you'd need to actually diagnose the cause. And in a distributed system, neither logs nor metrics from any single service show you the whole story of one user-facing request that hopped across five different services — each service's own logs only show its own piece, in isolation, with no obvious way to stitch the pieces back together into one coherent narrative.

The Solution — Three Complementary Signals, Not One

Logs, metrics, and traces aren't competing options — they're complementary, each covering the others' blind spots. A metric spike tells you something's wrong and roughly when; a trace shows you exactly which downstream call in one affected request was actually slow or failing; logs at that point in that trace give you the precise detail — the exception message, the specific input — needed to actually fix it. Together, they let you go from "something is wrong" to "here's exactly what, where, and why" without ever needing to reproduce the problem locally.

Big Picture

A user's checkout request hits your system:

  API Gateway → OrderService → PaymentService → InventoryService
        │              │              │                │
      LOGS           LOGS           LOGS              LOGS      (each service's own event record)
        │              │              │                │
        └──────────────┴──────────────┴────────────────┘
                            │
                        ONE TRACE
              (the whole journey, stitched together,
               showing exactly where the 4-second delay was)

Meanwhile, across ALL requests over the last hour:

                        METRICS
        (p99 latency, error rate, requests/sec —
         "is this normal, or is something degrading?")

A trace answers "what happened on this one journey." Metrics answer "what's the overall pattern across everything." Logs answer "what exactly happened at this one point." None of the three substitutes for the others.

How It Works

Logs — Already Familiar Territory

You already know this pillar in depth: ILogger<T> and structured logging with named message templates, from earlier in this course. Every log entry is a discrete record — "at this timestamp, this event happened, with these structured properties attached." That's exactly the raw material observability's log pillar is built from; nothing new to learn here mechanically, just a new frame for why it matters at this scale.

Metrics — Answering "How Much/Many/Fast," Efficiently

A metric is a number, tracked over time, usually aggregated: a counter (total requests handled), a gauge (current queue depth right now), or a histogram (the distribution of request latencies, letting you ask for the p50, p95, p99 — "95% of requests finished faster than X"). Metrics are deliberately compact — instead of storing every individual event, they store aggregated statistics, which is exactly why they scale to answering "how is the whole system doing right now" without drowning in raw event volume the way scanning every log line would.

Traces — Following One Request's Actual Journey

A trace represents one logical operation — typically one incoming request — as it flows through your system, made up of individual spans: one span for the time spent in the API gateway, another for the call to OrderService, another nested inside it for the call to PaymentService, and so on. Every span carries a shared identifier tying it back to the same overall trace, so a tracing tool can reconstruct the entire journey — in order, with exact timing — even though the work happened across several completely separate processes.

Simple Example

The same slow checkout, seen through each of the three pillars:

METRIC (aggregated, across all checkouts, last 15 minutes)
  checkout.duration.p99 = 4200ms   ← up from a usual ~800ms. Something is wrong, system-wide.

TRACE (one specific slow checkout request)
  ├─ API Gateway                     12ms
  ├─ OrderService.CreateOrder        45ms
  │    └─ PaymentService.Charge    4010ms  ← here. This span is where the time went.
  └─ InventoryService.Reserve        38ms

LOG (from inside PaymentService, at that exact span)
  warn: PaymentService[0]
        Payment gateway retry 3/3 for OrderId 88213 — upstream timeout

The metric told you that something changed. The trace told you exactly where in the request the time went — one specific downstream call, not the whole pipeline. The log, scoped to that exact span, told you why — the payment gateway itself was timing out and being retried. No single pillar gets you from "something's wrong" to "here's the root cause" alone.

Real-World Example

Consider an on-call engineer paged at 2am because an alert fired: error rate on the checkout endpoint crossed 5% (a metric threshold). Without traces, they'd have to guess which of a dozen downstream services is involved, and start manually grepping through each service's own logs hoping to spot a pattern — slow, and error-prone under time pressure. With tracing wired in, they instead open a handful of the actual failed traces from that time window, immediately see that every one of them failed inside the same downstream InventoryService call, follow that span directly to the exact log lines emitted at that failure point, and see the actual exception: a specific product ID causing a null-reference deep in inventory lookup logic. What could have been an hour of guessing becomes a few minutes of following a direct thread from symptom to cause — this is the entire practical payoff of having all three pillars in place before you need them, not after.

Analogy

A Hospital's Three Kinds of Records

Metrics are like a hospital's dashboard of vital signs across the whole building right now — average wait time, beds occupied, admissions per hour. It tells staff "something's off in the ER tonight" without describing any one patient.

Traces are like one specific patient's full visit record — check-in, triage, which doctor saw them, which department they were sent to, how long each step took, in order. It reconstructs one patient's entire journey through the building.

Logs are like the individual notes a nurse or doctor writes at each specific step — "10:42pm, patient reports chest pain, ordered ECG." Detailed, timestamped, but only about one moment.

You'd never run a hospital on just one of these. The vitals dashboard tells you to look closer; the patient's visit record tells you where in their visit to look; the doctor's notes tell you exactly what was found there.

Under the Hood

WHY YOU CAN'T "JUST DEBUG IT" IN PRODUCTION

1. A LOCAL DEBUGGER PAUSES ONE PROCESS, ON DEMAND

Stepping through code assumes you can freeze execution and inspect state — reasonable for one process handling one request on your own machine, with nobody else depending on it staying responsive.

▼

2. PRODUCTION HAS MANY PROCESSES, SERVING REAL TRAFFIC, CONCURRENTLY

Pausing any one instance to inspect it either does nothing useful (the bug is timing- or load-dependent and won't reproduce under a debugger) or actively causes the outage you were trying to diagnose, by refusing traffic while paused.

▼

3. SO THE SYSTEM HAS TO RECORD ITS OWN BEHAVIOR AS IT HAPPENS

Observability shifts debugging from "pause and inspect live" to "the system already recorded enough about what it did, and I query that record after the fact." This is why logs, metrics, and traces are emitted continuously, as a normal part of running the app — not turned on only when something's already broken.

Common Confusion

"Monitoring and observability are the same thing" — related, not identical

Monitoring usually means watching a known set of predefined signals for known failure modes — "alert me if CPU exceeds 90%." Observability is broader: it's about having enough recorded detail to answer questions you didn't think to ask in advance, when a genuinely new, unanticipated kind of problem shows up. Good monitoring tells you something's wrong; genuine observability lets you actually figure out why, even for a failure mode nobody explicitly built a dashboard for.

"More logging is basically the same as observability" — not once you're distributed

Heavy, detailed logging feels like it should be enough — and for a single-process app, it often mostly is. But logging alone doesn't give you cheap aggregate answers at scale (that's what metrics are for), and it doesn't natively stitch together one request's path across multiple separate services (that's what traces are for). Once a single user-facing request can touch several different processes, log volume alone stops being a substitute for the other two pillars.

Common Mistakes

Mistake 1 — Relying on logs alone once the system is distributed

Trying to reconstruct a multi-service request's journey by manually correlating timestamps across several services' separate log streams — slow, error-prone, and it doesn't scale past a handful of services.

Add tracing so the request's journey is captured directly, with an explicit shared identifier tying every service's contribution to the same one trace.

Mistake 2 — Adding observability only after the first painful incident

Treating logs, metrics, and traces as something to bolt on retroactively, once a production issue has already been painful to diagnose without them.

Wire up all three pillars as a normal part of building the system, before you need them — you can't retroactively generate a trace for a request that already finished and wasn't being traced at the time.

Mistake 3 — Treating a metric spike as the diagnosis, not just the symptom

Seeing "error rate spiked" and stopping there, without following it down into a trace and the relevant logs to find the actual root cause.

Use metrics to know that and roughly when something's wrong, then use traces and logs to find out why — each pillar plays its own role in that chain.

When Should I Use It?

Structured logging is worth having in essentially any real application beyond a throwaway script — you already know why, from earlier in this course.
Metrics become genuinely valuable once you care about aggregate system health — latency trends, error rates, capacity — rather than just individual events.
Tracing becomes especially, non-negotiably valuable the moment a single logical request can flow across more than one service or process — a monolith with no external service calls has much less need for it than a system split across several deployable pieces.
A tiny personal project with no real traffic and no distributed calls doesn't need the full weight of this — but it's worth understanding the pillars before you need them, because retrofitting observability into an already-struggling production system under time pressure is far harder than building it in from the start.

Looking ahead: Knowing the three pillars is half the story — actually implementing all three, wired together consistently, is the other half. The next lesson, OpenTelemetry, covers the modern, standard way to implement logs, metrics, and traces together under one unified system.

Mental Model

Logs = what happened, at this one moment
Metrics = how much/many/fast, aggregated over time
Traces = the full journey of one request, across everything it touched

Remember: metrics tell you something's wrong; traces tell you where; logs tell you why. You need all three because you can't pause a live, distributed system to look at it directly.

Key Takeaway

Observability is the ability to understand a running system's internal behavior purely from its external signals — because you can't pause and step through a live production system serving real, concurrent traffic.
Logs are discrete, timestamped event records — you already know these via ILogger<T> and structured logging.
Metrics are numeric measurements aggregated over time — request counts, latency percentiles, error rates — answering "how much/many/fast" efficiently at scale.
Traces follow one request's actual journey across services and components, especially necessary once a system is distributed.
The three pillars are complementary, not redundant — each covers a blind spot the others have.
Observability matters more, not less, as systems grow more distributed — logs/metrics/traces are the debugging tools available to you once "attach a debugger" stops being an option.

Check Your Understanding

You've seen why production systems demand a different kind of debugging. Let's confirm you understand how the three pillars divide the work.

1. An on-call engineer wants to know: "across the last hour, what's the 99th-percentile response time for the checkout endpoint?" Which pillar is the right tool for this specific question?

A) Logs — scan every individual log line from the last hour and compute it manually
B) Metrics — a histogram already aggregates response times, so the p99 can be read directly without scanning individual events
C) Traces — follow one single request's journey to determine the answer
D) None of the three pillars can answer aggregate questions like this

Show answer

Correct: B

Why B is correct: This is exactly the kind of "how much/how fast, in aggregate" question metrics are built for — a latency histogram is already computing percentiles across many events, making this a cheap, direct lookup rather than a scan through raw data.

Why A is incorrect: Logs could theoretically be used to compute this, but doing so at scale is exactly the inefficient approach metrics exist to avoid.

Why C is incorrect: A single trace shows one request's journey, not an aggregate statistic across an hour of many requests.

Why D is incorrect: Metrics are specifically designed to answer aggregate questions like this efficiently.

Reinforcement: Reach for metrics whenever the question is about volume, rate, or distribution across many events, not any single one.

2. A single user-facing request passes through an API gateway, an order service, and a payment service before returning. One specific request was unusually slow. Which pillar is specifically designed to show exactly which of those three hops the delay occurred in?

A) Logs alone, by manually cross-referencing timestamps across all three services' separate log streams
B) Metrics, since aggregate latency percentiles will point to the specific slow hop
C) A trace, which follows this one specific request across all three services with per-hop timing, tied together by a shared identifier
D) None of the pillars can show this — you would need to reproduce the issue locally with a debugger

Show answer

Correct: C

Why C is correct: This is exactly a trace's job — reconstructing one specific request's path across multiple services, with per-span timing, using a shared identifier that ties each service's contribution back to the same overall journey.

Why A is incorrect: This is technically possible but exactly the slow, error-prone, non-scaling approach the lesson describes as the reason tracing exists in the first place.

Why B is incorrect: Metrics show aggregate trends across many requests, not the specific path of one particular request.

Why D is incorrect: Reproducing a distributed, timing-dependent issue locally is often impractical or impossible — this is precisely why observability tooling exists as an alternative to live debugging.

Reinforcement: Traces are the pillar purpose-built for "where, specifically, in this one request's journey did the problem happen."

3. Why does observability become more important, not less, as a system becomes more distributed across many services and instances?

A) Distributed systems are actually easier to debug with a traditional debugger than monolithic ones
B) You cannot pause or step through a live, distributed system serving real concurrent traffic the way you can step through local code — recorded logs, metrics, and traces become the primary way to understand what happened
C) Observability is only relevant for systems with a single running instance
D) More services automatically means fewer possible failure points to monitor

Show answer

Correct: B

Why B is correct: This is the lesson's central point — a distributed, live production system can't be paused and inspected the way local code can be stepped through, and a single request may span multiple processes with no one place to directly "watch" it happen. Logs, metrics, and traces become the tools that fill that gap.

Why A is incorrect: Distributed systems are generally harder, not easier, to debug with a traditional live debugger — there's no single process to attach to that captures the whole picture.

Why C is incorrect: Observability matters for single-instance systems too, but its necessity grows sharply once requests span multiple services — the opposite of this option's claim.

Why D is incorrect: More services generally means more potential failure points and more places a request can go wrong, not fewer — this is part of why distributed systems need stronger observability, not weaker.

Reinforcement: The less directly you can observe a system live, the more its own recorded telemetry has to do the work instead.

4. Which statement correctly distinguishes monitoring from the broader idea of observability, as covered in this lesson?

A) They are exactly the same thing, just different names for identical practices
B) Monitoring typically watches a predefined set of known signals for known failure modes, while observability is the broader ability to answer previously unanticipated questions using recorded telemetry
C) Monitoring only applies to metrics, while observability only applies to logs
D) Observability is a marketing term with no real technical distinction from monitoring

Show answer

Correct: B

Why B is correct: Monitoring is generally about watching for known, predefined conditions ("alert if X exceeds Y"). Observability is broader — having enough recorded, queryable detail to diagnose problems you didn't specifically anticipate or build a dashboard for in advance.

Why A is incorrect: They're related but meaningfully distinct, as described above — treating them as identical misses the actual point of the distinction.

Why C is incorrect: Both monitoring and observability can draw on logs, metrics, and traces — the distinction is about known-vs-unanticipated questions, not which pillar each uses.

Why D is incorrect: The distinction, while sometimes overused in marketing contexts, reflects a genuine, real difference in what each practice lets you do.

Reinforcement: Good monitoring catches known problems; genuine observability lets you diagnose the ones nobody thought to specifically watch for.

You now understand the three pillars of observability and why distributed systems make them essential, not optional. Next: OpenTelemetry, the modern, standard way to implement all three together.