Structured Logging Across Services: Correlated .NET Logs
OrderFlow's order pipeline touches five services on a good day and needs to explain itself on a bad one. Lesson 128 taught you how to log — this lesson is about what OrderFlow actually says, stage by stage, and what it knows better than to say at all.
Welcome to the back half of Part XIII. Lessons 333 through 339 assembled OrderFlow's skeleton: Clean Architecture layering (333-334), JWT authentication (335), an EF Core-backed database behind a Repository Pattern (336), an IDistributedCache in front of the product catalog (337), hosted background services processing payment, inventory, and shipping (338), and a Kafka-based messaging pipeline with the Outbox pattern publishing an OrderPlaced event on every checkout (339). That's a real system now — five independently deployed pieces cooperating to turn a checkout request into a shipped order. The next seven lessons don't add new pieces. They make the system you already built observable, testable, deployable, fast, and shippable — the difference between "it works on my machine" and "it works, and I can prove it, in production."
This lesson starts with logging, and it isn't going to re-explain ILogger<T>, structured message templates, or log levels — lesson 128 already did that thoroughly. Instead, you'll walk OrderFlow's actual order pipeline stage by stage, deciding exactly what gets logged at each hop, at what level, and — just as importantly — what never gets logged at all.
What Is It?
The Simple Explanation
Applied logging for OrderFlow means deciding, for each stage an order passes through — API request, validation, persistence, event publish, payment, inventory, shipping, notification — exactly one thing worth logging and exactly one severity that event deserves, using the ILogger<T> mechanics lesson 128 already taught. It also means drawing a hard, explicit line around what OrderFlow's logs are never allowed to contain.
The Technical Definition
OrderFlow's logging strategy attaches a structured, per-order logger.BeginScope carrying OrderId at the moment an order enters the pipeline, so every downstream log line — from OrderService through PaymentService, InventoryService, and ShippingService, even after crossing a Kafka topic boundary — can be correlated back to one order without manually threading an ID through every method signature. Each stage logs at the level matching what actually happened (Information for expected transitions, Warning for recoverable anomalies, Error for failures needing attention), and a small, explicit denylist — full card numbers, CVVs, raw payment tokens, passwords — is enforced by never passing those fields into a log call in the first place, not by trying to scrub them afterward.
Why Does It Exist?
The Problem — One Order, Five Services, No Single Log File
Before lesson 339's messaging pipeline existed, a broken checkout meant reading through one service's logs. Now an order genuinely lives across five independently deployed pieces: the API that accepted it, the background outbox publisher that announced it, and three separate consumers that react to it asynchronously. If each service logs whatever seems locally reasonable, with no shared identifier and no consistent sense of what "Warning" versus "Error" means, then a customer support ticket saying "my order never shipped" turns into a scavenger hunt across five sets of logs that don't obviously belong to the same story.
The Solution — One Correlated Story, Deliberately Told
The fix is exactly the discipline lesson 128 already gave you, applied consistently everywhere: structured templates so OrderId is a real, queryable field and not buried in a sentence, and BeginScope so it doesn't have to be re-typed into every single log call. Applied across all five of OrderFlow's services, with a consistent level convention, a support ticket about one order becomes one query — "show me every log line with OrderId = 8842, across every service, in the last hour" — instead of five separate manual searches, guessing at how each team phrased things.
Big Picture — OrderFlow's Pipeline, Stage by Stage
Here's the whole pipeline from lessons 336-339, with the logging decision made explicit for each stage:
| Stage | What's logged | Level |
|---|---|---|
| API receives checkout request (335's JWT auth already validated the caller) | Order received, CustomerId, item count | Information |
| OrderService persists the Order/OrderItem rows (336) | Order persisted, OrderId assigned | Information |
| Outbox row committed in the same transaction (339) | Outbox message queued for OrderPlaced | Debug |
| Background publisher (338) ships the outbox row to Kafka | Event published, topic + partition + offset | Information |
| PaymentService consumes and charges | Charge attempted; charge succeeded / declined / gateway timeout | Information (success) / Warning (declined, retryable) / Error (gateway unreachable) |
| InventoryService reserves stock | Stock reserved; or insufficient stock for a line item | Information / Warning |
| ShippingService schedules the shipment | Shipment scheduled with carrier | Information |
| NotificationService emails the customer | Confirmation email sent; or send failed, will retry | Information / Warning |
Notice the pattern: the expected happy path is always Information, a recoverable anomaly (a declined card that the customer can retry, a stock shortfall that triggers backorder handling) is Warning, and only a failure that genuinely needs a human's attention — a payment gateway that's unreachable entirely, not just declining one card — earns Error.
How It Works — Following One OrderId Across Five Services
using var _ = logger.BeginScope("OrderId:{OrderId}", order.Id);— every log line for the rest of this request automatically carriesOrderId, without re-typing it.
- The scope can't cross a Kafka topic boundary — a new process, a new scope. So the event payload itself carries
OrderId, and every consumer opens its ownBeginScopefrom that same value the instant it deserializes the message.
- Four independent processes, four independent log streams — but every single line in all four carries the same
OrderIdvalue, because each one deliberately re-establishes the scope rather than inventing its own correlation scheme.
- One query for
OrderId = 8842, across every service's aggregated output, returns the order's complete life story in chronological order — placed, charged, stock reserved, shipped, notified — even though it was never written by a single process.
Simple Example
public class PaymentService(IPaymentGateway gateway, ILogger<PaymentService> logger)
{
public async Task<ChargeResult> ChargeAsync(OrderPlacedEvent evt, CancellationToken ct)
{
using var _ = logger.BeginScope("OrderId:{OrderId}", evt.OrderId);
logger.LogInformation("Charging {Amount:C} for order", evt.Total);
// NEVER log evt.PaymentMethod.CardNumber, CVV, or the raw gateway token —
// only an identifier the gateway itself considers safe to echo back.
var result = await gateway.ChargeAsync(evt.PaymentMethod, evt.Total, ct);
if (result.Succeeded)
{
logger.LogInformation(
"Charge succeeded, gateway reference {GatewayReference}",
result.GatewayReference);
}
else if (result.IsRetryable)
{
logger.LogWarning("Charge declined but retryable: {Reason}", result.DeclineReason);
}
else
{
logger.LogError("Charge failed permanently: {Reason}", result.DeclineReason);
}
return result;
}
}Meaning: The scope makes OrderId free on every line inside the method. The level branches on what actually happened, not on how the code felt to write. And the card number never enters a log call anywhere — result.GatewayReference is an opaque identifier the payment gateway itself hands back specifically so a merchant can reference a transaction without ever holding the underlying card data.
Real-World Example — Reading Lesson 321's Incident With Better Logs
Lesson 321 walked through a checkout API timing out — traced, in the end, to a dropped .Include() causing an N+1 query that starved the thread pool. That whole investigation leaned on distributed tracing and diagnostic tooling because the logs alone didn't tell the full story. Applied logging is what prevents needing quite that much archaeology next time: if OrderService logs "Order persisted" at Information with a timestamp, and the next expected line — "Charge attempted" from PaymentService — doesn't show up in the same OrderId trail for several seconds where it used to take milliseconds, that gap is visible directly in the logs, before anyone needs to reach for a profiler. Good applied logging doesn't replace lesson 315's diagnostic toolkit — it's the first, cheapest signal that something in the pipeline is worth diagnosing at all.
Analogy
A package shipped across a courier network passes through a sorting facility, a regional hub, a delivery truck, and a final scan at the door — four different physical locations, four different systems recording activity. What makes tracking actually useful isn't that each location keeps meticulous internal notes; it's that every single scan, at every location, is tagged with the same tracking number. Search that one number and the courier's system assembles the whole journey for you, across facilities that never directly talked to each other. OrderId, carried through every scope across OrderFlow's five services, is that tracking number — the correlation is what makes five separate logs read as one story.
Under the Hood — Why the Denylist Has to Be Enforced at the Call Site
It's tempting to imagine a downstream log-scrubbing step that redacts anything that looks like a card number before it's stored. Don't rely on that as the primary control. A regex-based scrubber can miss a format it wasn't written for, and by the time it runs, the raw value may have already passed through a log aggregation pipeline, a message queue, or a third-party log shipping agent that persisted it before the scrubber ever saw it. The only control that's actually reliable is the one lesson 128's Mistake 2 already named: never construct a log call — structured or otherwise — with a sensitive field as an argument in the first place. evt.PaymentMethod in the example above is deliberately never passed to a log call directly; only result.GatewayReference, a value the payment gateway itself designed to be safely echoed back, ever appears in a log line.
Common Confusion
1. "A declined card is an Error" — no, it's an expected, routine outcome
A card getting declined is one of the most common, entirely normal outcomes a payment system handles — customers mistype numbers, cards expire, banks flag unusual purchases. Logging every decline at Error level means PaymentService's error rate looks catastrophic on a completely ordinary day, and it trains whoever's watching the dashboard to ignore Error-level alerts because they fire constantly for nothing actionable. A decline is Warning — genuinely worth recording, not a page-the-on-call-engineer event.
2. "If it's not a password, it's safe to log" — payment data is a whole other category
Lesson 128's denylist example centered on passwords, and it's easy to assume that's the entire category of things to worry about. A card number, a CVV, a raw payment token — none of these are passwords, and all of them are exactly as dangerous to leak. Any value that, on its own, could be used to move money or impersonate a payment method belongs on the denylist, not just credential-shaped values.
Common Mistakes
Mistake 1 — Logging the whole event object with a generic serializer call
logger.LogInformation("Processing {@Event}", evt) where evt is the full OrderPlacedEvent, including the nested PaymentMethod — a structured logging provider that supports object destructuring will happily serialize every field, card number included, without anyone intending it.
Log specific, named fields explicitly — never the whole object — so a new field added to PaymentMethod next month can't silently start leaking into logs.
Mistake 2 — Letting each of OrderFlow's five services invent its own correlation field name
OrderService logs OrderId, but ShippingService logs the same value as order_id or RelatedOrder — a cross-service query now needs to know every service's private naming convention.
Agree on one field name, used identically in every scope across every one of OrderFlow's services — the whole benefit of correlation collapses the moment the field names diverge.
Mistake 3 — Logging at Information for every step of a high-volume, low-value hop
Logging every single outbox row scanned by the background publisher (338) at Information, even the ones with nothing new to publish — on a busy day this drowns the genuinely interesting Information-level events in noise nobody reads.
Route routine, high-frequency, low-signal detail to Debug or Trace, reserving Information specifically for events someone actually wants to see scrolling by in production.
When Should I Use It?
- Open an OrderId scope the instant an order exists, in every service that will ever touch it — not just where it's convenient.
- Reserve Error for failures that need a human's attention right now; Warning for expected, recoverable anomalies; Information for the pipeline's normal, expected transitions.
- Treat the sensitive-data denylist as a call-site discipline, not a downstream cleanup step — never construct the log call with the sensitive value in the first place.
Mental Model
Level = how loudly this specific outcome deserves to be announced — not how the code felt to write
Denylist = enforced at the log call itself, never as an afterthought cleanup step
Remember: lesson 128 gave you the mechanics; this lesson is the five-service, one-OrderId discipline that makes those mechanics actually pay off.
Key Takeaway
- Every one of OrderFlow's five pipeline stages — API intake, persistence, outbox publish, payment, inventory, shipping, notification — has a deliberate, specific log level, chosen to match reality, not convenience.
- A single, consistently-named OrderId scope, re-opened in every service the order touches, turns five separate log streams into one reconstructable story.
- Payment details never become log call arguments in the first place — the denylist is enforced at the call site, not as a downstream scrubbing step that could miss something or run too late.
- Applied logging is the cheapest first signal something's wrong in the pipeline — it doesn't replace lesson 315's diagnostic toolkit, but it's often what tells you a deeper investigation is warranted at all.
Check Your Understanding
You've seen how OrderFlow's pipeline turns lesson 128's mechanics into an actual, correlated, safe logging strategy. Let's confirm it clicked.
1. A customer's card is declined because they mistyped a digit. What level should PaymentService log this at?
Show answer
Correct: B
Why B is correct: A mistyped-digit decline is one of the most routine outcomes a payment system handles — Warning captures "worth recording, recoverable" without triggering the same alerting a genuine gateway outage should.
Why A is incorrect: Logging every ordinary decline at Error trains whoever watches the error rate to ignore it, since it fires constantly for a completely normal event.
Why C is incorrect: A decline is a real business event worth being able to find later (for support, for fraud analysis) — Debug-level noise is easy to lose entirely in production.
Why D is incorrect: Critical is reserved for failures threatening the whole application's operation — one customer's mistyped card number doesn't rise anywhere near that.
Reinforcement: Match the level to what actually happened, not to how alarming payment failures sound in the abstract.
2. Why does NotificationService need to re-open an OrderId scope, rather than simply inheriting the scope OrderService opened earlier?
Show answer
Correct: B
Why B is correct: A logging scope lives entirely within one process's in-memory logging pipeline — it has no way to travel across a message broker. The event payload carrying OrderId as a plain field is exactly what lets each independent consumer re-establish the same correlation on its own.
Why A is incorrect: Scopes are explicitly local to the process that created them — there's no built-in cross-process propagation, which is precisely why the event payload needs to carry the identifier itself.
Why C is incorrect: Scopes don't expire on a timer — they're tied to the lifetime of the using block that created them, within one process.
Why D is incorrect: Every service in the pipeline opens its own scope from the same OrderId value — that's exactly what keeps the whole trail correlated across all five services.
Reinforcement: Correlation across service boundaries needs the identifier to travel in the data itself (the event payload), not in an in-memory mechanism that can't leave one process.
3. Why is scrubbing sensitive fields out of logs after the fact — with a downstream regex-based redaction step — not a reliable substitute for never logging them in the first place?
Show answer
Correct: B
Why B is correct: This is exactly the Under the Hood reasoning — a scrubber is a pattern match that can fail to recognize an unexpected format, and by the time it would run, the raw sensitive value may have already been persisted or forwarded by infrastructure upstream of the scrubber.
Why A is incorrect: Performance isn't the reasoning given — reliability and timing (the value already being persisted elsewhere) are the actual concerns.
Why C is incorrect: Downstream log processing is entirely possible in .NET and common in real systems — it's just not trustworthy as the *only* safeguard against leaking sensitive data.
Why D is incorrect: The lesson explicitly recommends the opposite — enforcing the denylist at the call site, not relying on downstream scrubbing as the primary control.
Reinforcement: The only reliable control is never constructing the log call with the sensitive value as an argument — everything downstream is too late to fully trust.
Next up: lesson 341 takes the correlated story this lesson's logs now tell and adds the metrics and traces that answer a different question — not "what happened to this one order," but "how is the whole system doing, right now, across every order."