Lesson 340's logs tell you what happened to one order. Monitoring tells you whether the whole system is healthy right now — across every order, before any single customer has to file a ticket about it.
Lesson 270 gave you OpenTelemetry: auto-instrumentation for ASP.NET Core, HttpClient, and EF Core, wired up in a few lines, producing connected traces with zero manual span creation. Lesson 303 gave you the cloud-native reality underneath it — stdout-based logging, structured JSON, and why OpenTelemetry matters even more once "SSH into the server" stops being a coherent idea. Both lessons were deliberately generic, true of any ASP.NET Core service. This lesson is the opposite: it's specifically about OrderFlow, and specifically about the handful of signals that actually tell you whether OrderFlow, as a whole, is doing its job.
Auto-instrumentation gives you a great trace of one HTTP request and one database query. It has no idea what "order success" means, what a "checkout" is, or that a payment declining is a business event worth its own counter. This lesson builds the custom metrics and traces OrderFlow needs on top of what 270's auto-instrumentation already gives you for free, and shows what they'd actually look like assembled on a dashboard.
Monitoring OrderFlow means picking a small number of business-meaningful signals — order success rate, checkout latency, payment failure rate, how far behind the outbox publisher and Kafka consumers are running — and emitting them as OpenTelemetry metrics and traces alongside the generic HTTP/database telemetry lesson 270's auto-instrumentation already produces, so a dashboard can answer "is OrderFlow healthy right now" at a glance.
OrderFlow instruments its own domain events using .NET's Meter and ActivitySource primitives — the same primitives lesson 270 showed OpenTelemetry itself is built on — to emit custom counters (orders placed, payments failed, tagged by outcome), histograms (checkout duration, in milliseconds), and a custom Activity spanning the full asynchronous pipeline from HTTP request through Kafka consumption. Trace context propagates across the Kafka boundary via message headers, so one trace can show a checkout request's synchronous HTTP portion and its asynchronous payment/inventory/shipping portion as a single, connected picture, instead of two unrelated fragments.
Lesson 270's auto-instrumentation is genuinely excellent at what it covers: it'll tell you the P99 latency of POST /api/orders, and it'll tell you EF Core's query duration. Neither of those is the number that actually matters to the business. A checkout that returns HTTP 200 in 80ms but whose downstream payment charge silently fails ten seconds later is, from the auto-instrumented HTTP span's point of view, a fast, successful request — and from the customer's point of view, an order that never shipped. Worse, OrderFlow's real "did this work" question spans multiple processes: the HTTP response happens before PaymentService, InventoryService, and ShippingService have even started their asynchronous work (338). No single auto-instrumented span covers that whole journey.
The fix isn't to abandon auto-instrumentation — it's to add a thin, deliberate layer of custom telemetry on top of it, exactly as lesson 270's Real-World Example already showed: a custom ActivitySource span nests naturally inside the automatically-generated request span, because it's built on the same underlying mechanism. OrderFlow adds a handful of counters and histograms tied to what "success" actually means for an order-management system, and propagates one trace context across the Kafka boundary so the async half of the pipeline shows up connected to the sync half, not as an orphaned fragment.
POST /api/orders duration — the HTTP-visible part of the experiencePaymentService/InventoryService/ShippingService's consumer group — the async pipeline's own backlog signalIDistributedCache (337) hit ratio — a falling hit rate is often the earliest sign of a coming database load spike_ordersPlaced.Add(1), _checkoutDuration.Record(elapsedMs), _paymentsFailed.Add(1, new("reason", declineReason)) — recorded at the exact moment each of those things actually happens in OrderService/PaymentService.reason; consumer lag as a per-service gauge — each one a direct visualization of one metric this lesson defined.public class OrderMetrics
{
private readonly Counter<long> _ordersPlaced;
private readonly Counter<long> _paymentsFailed;
private readonly Histogram<double> _checkoutDuration;
public OrderMetrics(IMeterFactory meterFactory)
{
var meter = meterFactory.Create("OrderFlow.Orders");
_ordersPlaced = meter.CreateCounter<long>("orderflow.orders.placed");
_paymentsFailed = meter.CreateCounter<long>("orderflow.payments.failed");
_checkoutDuration = meter.CreateHistogram<double>("orderflow.checkout.duration", unit: "ms");
}
public void RecordOrderPlaced() => _ordersPlaced.Add(1);
public void RecordPaymentFailed(string reason) =>
_paymentsFailed.Add(1, new KeyValuePair<string, object?>("reason", reason));
public void RecordCheckoutDuration(double elapsedMs) =>
_checkoutDuration.Record(elapsedMs);
}Meaning: This is a thin, deliberate wrapper — three signals, each tied to a specific business moment, not a generic "log everything" instrument. reason as a tag on _paymentsFailed is what lets the dashboard break payment failures down by cause (declined, gateway timeout, invalid card) without needing a separate counter per reason.
The payment failure rate tile, normally flat around 3-4% (ordinary declines), starts climbing toward 40% over ten minutes, broken down by reason=GatewayTimeout specifically — not reason=CardDeclined. That single detail already rules out "customers are having a bad day with their cards" and points at the payment gateway itself. The on-call engineer opens one of the failing traces: the custom ActivitySource span from this lesson shows the HTTP checkout request completing normally, followed — connected, via the propagated trace context across Kafka — by a PaymentService span that's taking 30 seconds and timing out, instead of its usual 200ms. Cross-referencing that trace's ID against lesson 340's correlated OrderId logs confirms it: every failing order shows the exact same LogError("Charge failed permanently: {Reason}", "GatewayTimeout") line. The dashboard caught the pattern in minutes; the trace confirmed exactly where; the logs confirmed exactly why — three signals, three different questions, one incident understood well before a support ticket would have surfaced it.
A car's speedometer tells you one thing well: how fast you're going right now. A pilot needs more than that to fly safely — altitude, airspeed, fuel remaining, engine temperature, each instrument answering a different specific question, laid out together so a glance across the panel tells the whole story at once. Auto-instrumentation is the speedometer: useful, automatic, but only one number. This lesson's metrics are the rest of the panel — order success rate is altitude (are we actually where we're supposed to be), payment failure rate is engine temperature (is something about to fail), consumer lag is fuel remaining (how much runway is left before the backlog becomes a real problem). No single gauge tells the whole story; the panel does.
Lesson 270's auto-instrumentation works because ASP.NET Core, HttpClient, and EF Core all participate in .NET's built-in Activity propagation — a downstream HttpClient call automatically carries the current trace's ID forward via a standard HTTP header. Kafka has no equivalent built-in behavior in .NET the way HTTP does, so OrderFlow's outbox publisher has to do this deliberately: when it publishes the OrderPlaced event, it serializes the current Activity's trace context (its W3C traceparent value) into the Kafka message's headers, right alongside the event payload. Each consumer — PaymentService, InventoryService, ShippingService — reads that header back out and starts its own Activity as a child of that same trace, rather than starting a brand-new, disconnected one. This one small, deliberate step is what turns "five separate traces" into "one trace with five stages" — it doesn't happen automatically the way it does over HTTP, precisely because a message broker has no request/response cycle for .NET's instrumentation to hook into on its own.
This is the exact gap this lesson exists to close. Auto-instrumentation (270) is necessary but not sufficient — it tells you about HTTP requests and database queries, generically, for any ASP.NET Core app. It has no idea an "order" exists, let alone what "success" means for one. Business-meaningful metrics have to be added deliberately; they don't fall out of generic instrumentation for free.
Lesson 340's correlated logs answer "what exactly happened to this one order" — precise, but you need to already know which order to ask about. This lesson's metrics answer "how is the whole system doing right now, across every order" — aggregate, cheap to keep forever, but they can't tell you the specific reason on their own. Traces sit in between: "what path did this one specific request actually take across every service." All three matter; none substitutes for the other two.
_checkoutDuration.Record(elapsedMs, new("orderId", order.Id)) — every distinct order ID becomes a distinct time series in the metrics backend, and with thousands of orders a day, this quietly explodes storage and query cost, sometimes badly enough to make the backend itself unusable.
Reserve metric tags for low-cardinality dimensions worth slicing by (reason, paymentProvider, a handful of known values) — OrderId belongs on a trace or a log line (340), never as a metric tag.
Paging the on-call engineer whenever "payments failed" exceeds some fixed number per hour — that threshold is either too sensitive on a high-traffic day or useless on a quiet one.
Alert on the failure rate (failures ÷ attempts) relative to its own recent baseline — the same 40 failures mean something very different at 4% of attempts versus 40% of attempts.
Publishing the OrderPlaced event without carrying the current trace context in its headers — every consumer then starts a brand-new, disconnected trace, and a checkout's full story has to be manually reassembled from unrelated trace IDs.
Serialize the Activity's trace context into the message headers at publish time, and have every consumer resume it — exactly the deliberate step "Under the Hood" walked through.
You've seen how OpenTelemetry's generic auto-instrumentation gets layered with OrderFlow-specific business metrics and traces. Let's confirm it clicked.
1. Why isn't lesson 270's auto-instrumentation alone sufficient to monitor OrderFlow?
Correct: B
Why B is correct: This is the lesson's central problem statement — a fast, HTTP-200 checkout whose downstream payment charge fails asynchronously still looks like a clean success to the auto-instrumented HTTP span alone, which is exactly the gap custom, business-meaningful metrics close.
Why A is incorrect: Lesson 270 demonstrated auto-instrumentation working correctly — the issue isn't that it's broken, it's that it's generic by design.
Why C is incorrect: OpenTelemetry explicitly supports all three signal types — logs, metrics, and traces — as lesson 270 and this lesson both use.
Why D is incorrect: The lesson explicitly builds custom metrics ON TOP OF auto-instrumentation, not as a replacement for it.
Reinforcement: Generic instrumentation and business-specific metrics are complementary — one doesn't make the other unnecessary.
2. Why does OrderFlow's outbox publisher need to explicitly serialize trace context into Kafka message headers, when an outbound HttpClient call gets this "for free"?
Correct: A
Why A is correct: This is exactly the Under the Hood explanation — HTTP has a built-in header convention .NET's instrumentation hooks into automatically; Kafka has no such built-in hook for .NET, so the publisher has to serialize and forward the trace context itself.
Why B is incorrect: The whole point of this lesson's trace-context work is to make asynchronous, cross-process work part of the same trace — it's not limited to synchronous HTTP calls.
Why C is incorrect: Kafka message headers are a standard feature with no such fixed size limit — this isn't the reason given anywhere in the lesson.
Why D is incorrect: .NET's Activity API is precisely what supports this — it's HTTP's built-in participation in that API that's automatic, not that the API itself is missing.
Reinforcement: Context propagation is automatic only where a subsystem already participates in .NET's Activity conventions — Kafka doesn't, so it must be done by hand.
3. Why is tagging a metric with OrderId flagged as a mistake in this lesson?
Correct: B
Why B is correct: This is exactly Mistake 1 — a high-cardinality tag like a per-order identifier multiplies the number of distinct time series a metrics backend has to track, which is a real, sometimes severe operational cost, unlike a low-cardinality tag such as reason.
Why A is incorrect: There's no type restriction being described — the issue is cardinality (how many distinct values a tag can take), not data type validity.
Why C is incorrect: The lesson's own example tags _paymentsFailed with reason — metrics tags are explicitly supported and used, just not with high-cardinality values.
Why D is incorrect: OrderId is explicitly the right fit for a trace or a log line (per lesson 340) — the lesson steers it away from metrics specifically, not away from telemetry entirely.
Reinforcement: Reserve metric tags for low-cardinality dimensions; put per-instance identifiers like OrderId on logs and traces instead.
Next up: with logging and monitoring both applied, lesson 342 turns to a different question — how much of OrderFlow's own confidence in itself should come from a real test suite, and which parts of Part XI's testing toolkit fit which part of the codebase.
dotnetmadeeasy.com — Learn C# and .NET, the right way.