Scaling .NET Apps Horizontally: Why Statelessness Matters
Making one instance fast is a different problem from running twenty instances at once — and the second one changes what your code is allowed to assume.
Part IV taught you how to make a single running instance of your app handle load well — the ThreadPool (209) and the async patterns behind high-throughput services (221) are both, fundamentally, about squeezing more concurrent work out of one process. That's real, valuable work, and it has a ceiling: eventually, one process on one machine runs out of CPU cores, memory, or raw capacity, no matter how efficiently it's written.
Cloud-native scaling asks a different question entirely: not "how do I make this one instance faster," but "how do I run many instances of this app at once, and spread load across all of them." That shift changes more than deployment mechanics — it changes what your application code is even allowed to assume about itself.
In this lesson, you'll learn the difference between vertical and horizontal scaling, why statelessness is the real, non-negotiable prerequisite for horizontal scaling to work correctly, how auto-scaling decides when to add or remove instances, and why the distributed-systems patterns from Part IX stop being optional the moment more than one instance is running.
What Is It?
The Simple Explanation
When an app can't handle its current load, there are exactly two directions to go: make the one machine it's running on bigger, or run more copies of it side by side and split the work between them. The first is vertical scaling. The second is horizontal scaling.
The Technical Definition
Vertical scaling (scaling up) increases the CPU, memory, or other resources allocated to a single running instance of an application. Horizontal scaling (scaling out) increases the number of running instances of an application, with a load balancer distributing incoming traffic across all of them.
⬆ Vertical Scaling
- One instance, made bigger (more CPU/RAM)
- Simple mentally — no distribution logic needed
- Hard ceiling — the largest machine still has a limit
- Usually requires downtime or a restart to resize
- A single point of failure remains a single point of failure
↔ Horizontal Scaling
- Many instances, same size, running side by side
- Load balancer spreads traffic across all of them
- No hard single-machine ceiling — add more instances
- Elastic — scale out and back in as real load changes
- Requires the app to be stateless (this lesson's core point)
Why Does It Exist?
The Problem — Vertical Scaling Has a Ceiling, and No Elasticity
Vertical scaling is the intuitive first move — if the app is slow, give it a bigger machine. It works, for a while. But it runs into two hard limits. First, there's an actual physical/cost ceiling: at some point, no larger machine is readily available, or the ones that are become disproportionately expensive. Second, and just as important in a cloud-native world: vertical scaling isn't elastic. Resizing a running machine's CPU/RAM allocation typically requires real downtime or at least a restart — it's not something you do smoothly, automatically, in direct response to a traffic spike that might last five minutes and then vanish.
The Solution — Horizontal Scaling, Elastically
Horizontal scaling sidesteps both limits. There's no hard ceiling — you're not waiting for a bigger machine to exist, you're adding another ordinary-sized one, and you can generally keep doing that far past where vertical scaling would have stalled. And critically, it's elastic: starting a new container instance takes seconds, not a maintenance window, so the number of running instances can track real, moment-to-moment demand — scaling out when traffic spikes, and scaling back in once it subsides, without ever touching a machine that's already serving traffic. This is precisely why horizontal scaling is the dominant, preferred approach in cloud-native architecture, and why containers (covered earlier in this Part) are such a natural fit for it — a container image is designed to be started as many identical, disposable copies as needed.
Big Picture
Both matter, and they're not competing — a well-scaled cloud-native system does both: each individual instance is written to use its own resources efficiently (Part IV), and the system as a whole runs many of those efficient instances side by side (this lesson).
How It Works — Statelessness Is What Makes This Safe
Horizontal scaling only behaves correctly under one real, non-negotiable condition: the application must be stateless. A stateless instance keeps no data in memory that only it knows about between requests — every request should be servable correctly by any instance, not specifically the one that handled the user's previous request.
- Instance A stores something in its own process memory — session data, or an entry in its own
IMemoryCache(lesson 265) — that only Instance A knows about.
- There's no guarantee the same user's next request lands back on Instance A — a load balancer generally has no reason to "stick" a user to one instance, and shouldn't need to.
- The user's session appears to have vanished, or a cached value that Instance A already computed gets recomputed from scratch on Instance B — because Instance B's in-process memory is a completely separate, disconnected island from Instance A's.
This is exactly the problem lesson 265 already flagged directly: IMemoryCache is a fast, in-process cache scoped to one running instance, and "the moment your app scales to more than one instance behind a load balancer, each instance's IMemoryCache becomes its own island." The fix isn't to work around horizontal scaling — it's to externalize the shared state so every instance reads from and writes to the same place:
Instance-local state (breaks horizontal scaling)
- In-memory session state stored on one instance
IMemoryCacheentries only that instance knows about- Any "sticky" assumption about which instance a user talks to
Externalized, shared state (correct fix)
IDistributedCache(lesson 266) — one shared cache every instance reads and writes- Data-specific caching strategies (lesson 280) applied against that shared store
- Session/user state persisted in a shared database or distributed store, not process memory
Simple Example
The exact same product-lookup method, written two ways — one that quietly breaks under horizontal scaling, and one that doesn't:
// Breaks the moment you run more than one instance
public class ProductService(AppDbContext db, IMemoryCache cache)
{
public async Task<Product?> GetAsync(int id) =>
await cache.GetOrCreateAsync($"product:{id}", async entry =>
{
entry.SlidingExpiration = TimeSpan.FromMinutes(10);
return await db.Products.FindAsync(id);
});
// Each instance builds its own separate copy of this cache.
// A write that invalidates it on Instance A never reaches Instance B.
}
// Safe under any number of instances
public class ProductService(AppDbContext db, IDistributedCache cache)
{
public async Task<Product?> GetAsync(int id)
{
var cached = await cache.GetStringAsync($"product:{id}");
if (cached is not null)
return JsonSerializer.Deserialize<Product>(cached);
var product = await db.Products.FindAsync(id);
if (product is not null)
await cache.SetStringAsync($"product:{id}",
JsonSerializer.Serialize(product),
new DistributedCacheEntryOptions { SlidingExpiration = TimeSpan.FromMinutes(10) });
return product;
}
// Every instance reads and writes the same shared store (lesson 266) —
// it doesn't matter which instance served the previous request.
}Real-World Example — Scaling an Order/Payment API for a Flash Sale
An order/payment API normally runs as 3 instances, handling routine traffic comfortably. A flash sale is announced, and request volume climbs to 15x normal within minutes. Because the service is genuinely stateless — order data lives in the shared database, cart/session data lives in a distributed cache (266), nothing important lives only in one instance's process memory — the platform's auto-scaler can respond by simply starting more identical instances, and the load balancer immediately begins routing traffic across all of them. No instance needs to "know" it's now one of 20 instead of one of 3; each one just keeps handling whichever requests the load balancer sends it, exactly as before.
When the sale ends and traffic drops back to normal, the same mechanism scales the instance count back down — again, safely, precisely because no instance was the sole holder of anything another instance, or a future instance, might need.
Auto-Scaling — Deciding When to Add or Remove Instances
Horizontal scaling's elasticity is usually driven automatically, not by a human watching a dashboard and manually starting instances. An auto-scaler continuously watches one or more signals and adjusts the running instance count to match:
| Trigger | What it measures |
|---|---|
| CPU utilization | Average CPU usage across running instances — scale out when consistently high, scale in when consistently low |
| Memory utilization | Similar idea, applied to memory pressure instead of CPU |
| Request queue depth | How many requests are waiting to be handled — a direct signal that current capacity isn't keeping up |
| Custom application metrics | A domain-specific signal your app itself exposes — e.g. "pending orders in the processing queue" — driving scale decisions tailored to what actually matters for that specific workload |
Whichever signal drives it, the mechanism is the same at its core: the auto-scaler compares the current value against a configured threshold, and adds or removes running instances to bring the system back toward the target. This is only meaningfully safe to do automatically, without human oversight of every scaling event, because a stateless instance can be added or removed at any moment without anyone needing to migrate state off it first.
Analogy
Vertical scaling is like replacing a bank's one teller with a single, extraordinarily fast super-teller who can process transactions quicker than any normal teller. It helps — until you hit the physical limit of how fast one person can possibly work, no matter how skilled.
Horizontal scaling is like opening more teller windows, each staffed by an ordinary, interchangeable teller, with a line-management system directing each customer to whichever window is free — that's the load balancer. This scales much further, and you can open or close windows as the line gets longer or shorter throughout the day.
But it only works smoothly if every teller has access to the same shared record of each customer's account — the bank's central ledger, not a private notebook one specific teller keeps in their own drawer. If Teller 3 wrote your balance update in their own private notebook, and your next visit gets routed to Teller 7, Teller 7 has no idea what happened. That private notebook is exactly what in-process state and IMemoryCache become once you have more than one instance — the shared ledger is exactly what IDistributedCache and a shared database provide instead.
Under the Hood — Where Part IX's Patterns Suddenly Become Mandatory
This is the connection worth internalizing above everything else in this lesson. Part IX taught you idempotency (290), retries (291), and eventual consistency (295) as patterns for building resilient distributed systems. Once your application runs as more than one instance, those patterns stop being "distributed systems best practice you might reach for eventually" and become facts about how your own, single application already behaves in production, every day.
- A client retries a request that timed out (lesson 291) — but the retried attempt gets routed by the load balancer to a different instance than the one that received the original attempt. Both instances are perfectly healthy; neither one individually did anything wrong.
- Idempotency (290) — an idempotency key checked against a shared deduplication table every instance can see — is what makes "processed independently by two different instances" safe instead of catastrophic.
- Once state lives in a shared cache or database rather than any one instance's memory, there's inherently a brief window where different instances, or different reads of that shared store, might momentarily disagree — exactly the trade-off lesson 295 named and explained how to engineer around.
None of this is a single-instance concern. A single instance handling one request at a time never has to worry about "what if a different copy of me is handling a retry of this same logical operation right now." The moment there's more than one instance, that scenario isn't a rare edge case anymore — it's a routine, expected possibility every single day the system runs under real, retried, occasionally-duplicated traffic.
Common Confusion
1. "Scaling is just an infrastructure/DevOps concern" — statelessness is an application design decision
It's tempting to think horizontal scaling is entirely something the platform team configures, with no impact on application code. Statelessness proves that wrong: whether your app is safe to horizontally scale is decided by how the application itself is written — where it keeps state — not by any infrastructure setting alone.
2. "Part IV's throughput work and this lesson's scaling are the same topic" — they're complementary, not redundant
Making one instance efficient (209, 221) and running many instances (this lesson) solve different problems and both matter. An inefficient instance, horizontally scaled, just means many inefficient instances running side by side — you generally want both: efficient instances, run in the right quantity for current demand.
Common Mistakes
Mistake 1 — Reaching for a bigger machine as the default first move
Defaulting to vertical scaling whenever load grows, without considering elasticity — a bigger machine has to be provisioned in advance and generally can't shrink back down smoothly once the spike passes.
Default to horizontal scaling for anything with variable, unpredictable load — it's what actually tracks demand up and down automatically.
Mistake 2 — Assuming a retried request always lands on the same instance
Building request handling that implicitly assumes "if this fails and gets retried, it'll be the same instance handling it again" — a load balancer gives you no such guarantee.
Design every operation to be safely idempotent (290), regardless of which instance ends up handling any given attempt.
Mistake 3 — Leaving IMemoryCache in place "because it worked fine before we scaled out"
Not revisiting caching choices when moving from one instance to many — IMemoryCache doesn't fail loudly, it just quietly produces inconsistent results per instance.
Move genuinely shared cache data to IDistributedCache (266) the moment more than one instance is in play.
When Should I Use It?
- Use horizontal scaling as the default for any cloud-native service expected to face variable or growing load — it's elastic and has no realistic single-machine ceiling.
- Vertical scaling can still be the right, simpler answer for a small, predictable workload where the added complexity of load balancing and enforced statelessness genuinely isn't worth it yet.
- Any service you intend to horizontally scale must be made stateless first — this is a prerequisite to design for, not an afterthought to retrofit once instances start disagreeing with each other in production.
- The moment a service can run as more than one instance, treat idempotency (290), retries (291), and eventual consistency (295) as required, not optional, design considerations.
Mental Model
Horizontal scaling = more machines, load balanced, elastic
Statelessness = the price of admission for horizontal scaling to work correctly
Part IX's patterns = no longer optional the instant you have more than one instance
Before scaling out, ask: "if the very next request from this same user lands on a totally different instance, does everything still work correctly?" If the honest answer is no, fix that first — scaling out won't wait for you to notice.
Key Takeaway
- Vertical scaling = a bigger machine for one instance. Horizontal scaling = more instances of the same size, load balanced across them.
- Horizontal scaling is the dominant cloud-native approach — it's elastic, tracking real demand up and down, and has no hard single-machine ceiling.
- Statelessness is the real prerequisite — in-memory session state or a per-instance
IMemoryCache(265) breaks the moment a load balancer routes a user's next request to a different instance. UseIDistributedCache(266) and the strategies from lesson 280 instead. - Auto-scaling responds to CPU/memory utilization, request queue depth, or custom application metrics to add or remove instances automatically.
- The moment an app runs as more than one instance, idempotency (290), retries (291), and eventual consistency (295) become necessary — "the same request, handled independently by two instances" is now routine, not an edge case.
Check Your Understanding
You've seen why running many instances is a genuinely different challenge from making one instance fast. Let's confirm the reasoning holds up.
1. Why is horizontal scaling generally preferred over vertical scaling for a cloud-native service with unpredictable, spiky traffic?
Show answer
Correct: B
Why B is correct: Elasticity and the absence of a single-machine ceiling are exactly why horizontal scaling dominates cloud-native design — it can respond to a spike in minutes and scale back down just as easily.
Why A is incorrect: The opposite is true — horizontal scaling specifically requires the application to be stateless to work correctly; that's the core prerequisite this lesson covers.
Why C is incorrect: Vertical scaling remains fully supported; it's just a poorer fit for elastic, unpredictable load.
Why D is incorrect: Horizontal scaling specifically requires a load balancer to distribute traffic across the running instances — it doesn't eliminate the need for one.
Reinforcement: Elasticity and no hard ceiling are the concrete, technical reasons horizontal scaling wins for variable load — not vague "it's more modern" reasoning.
2. A service stores each logged-in user's shopping cart in an IMemoryCache entry. After the service is horizontally scaled to 4 instances behind a load balancer, users report their carts randomly appear empty. What's the root cause?
Show answer
Correct: B
Why B is correct: This is exactly the failure mode lesson 265 warned about and this lesson builds on — IMemoryCache is per-instance. A request landing on a different instance than the one that stored the cart simply won't find it there.
Why A is incorrect: There's no such cross-instance limit; the real issue is each instance has an entirely separate cache, not a shared one with a capacity problem.
Why C is incorrect: Load balancers route traffic; they don't inspect or delete application-level cache data.
Why D is incorrect: Scaling doesn't clear existing instances' caches — the problem is that each instance never had the other instances' data in the first place.
Reinforcement: The fix is IDistributedCache (266) — one shared store every instance reads and writes, so it doesn't matter which instance serves any given request.
3. Why do idempotency and retries (Part IX) become necessary, not just theoretically useful, once an application runs as multiple horizontally-scaled instances?
Show answer
Correct: B
Why B is correct: This is the lesson's central connection back to Part IX — once more than one instance exists, a retried request landing on a different, equally healthy instance is normal, everyday behavior, and idempotency is what keeps that from causing duplicate side effects like a double charge.
Why A is incorrect: Horizontal scaling is about capacity and elasticity, not per-request speed, and doesn't inherently slow anything down.
Why C is incorrect: Exception handling is unrelated to and unaffected by how many instances are running.
Why D is incorrect: Load balancers forward retried requests exactly like any other request — they have no special restriction against retries.
Reinforcement: More instances means "the same logical operation, handled independently more than once" stops being unlikely and starts being routine — design for it accordingly.
4. Which auto-scaling trigger would be most appropriate for a service where slow requests are best predicted by how many requests are currently waiting to be processed, rather than by raw CPU usage?
Show answer
Correct: B
Why B is correct: Request queue depth directly measures the exact symptom described — work piling up faster than it's being processed — which can be a more direct, earlier signal than CPU usage for I/O-bound or unevenly-loaded workloads.
Why A is incorrect: CPU utilization is a valid trigger in general, but the scenario specifically describes a case where it's a poorer fit than queue depth.
Why C and D are incorrect: Neither reflects real-time load or demand in any way — they're static, build-time properties with no bearing on current traffic.
Reinforcement: Different workloads are better served by different auto-scaling signals — CPU/memory, queue depth, or a custom application-defined metric — chosen based on what actually predicts the service falling behind.
You now understand what actually changes when an application moves from "efficient on one machine" to "running as many instances at once" — and why statelessness and the Part IX resilience patterns are what make that shift safe.