Making one instance fast is a different problem from running twenty instances at once — and the second one changes what your code is allowed to assume.
Part IV taught you how to make a single running instance of your app handle load well — the ThreadPool (209) and the async patterns behind high-throughput services (221) are both, fundamentally, about squeezing more concurrent work out of one process. That's real, valuable work, and it has a ceiling: eventually, one process on one machine runs out of CPU cores, memory, or raw capacity, no matter how efficiently it's written.
Cloud-native scaling asks a different question entirely: not "how do I make this one instance faster," but "how do I run many instances of this app at once, and spread load across all of them." That shift changes more than deployment mechanics — it changes what your application code is even allowed to assume about itself.
In this lesson, you'll learn the difference between vertical and horizontal scaling, why statelessness is the real, non-negotiable prerequisite for horizontal scaling to work correctly, how auto-scaling decides when to add or remove instances, and why the distributed-systems patterns from Part IX stop being optional the moment more than one instance is running.
When an app can't handle its current load, there are exactly two directions to go: make the one machine it's running on bigger, or run more copies of it side by side and split the work between them. The first is vertical scaling. The second is horizontal scaling.
Vertical scaling (scaling up) increases the CPU, memory, or other resources allocated to a single running instance of an application. Horizontal scaling (scaling out) increases the number of running instances of an application, with a load balancer distributing incoming traffic across all of them.
Vertical scaling is the intuitive first move — if the app is slow, give it a bigger machine. It works, for a while. But it runs into two hard limits. First, there's an actual physical/cost ceiling: at some point, no larger machine is readily available, or the ones that are become disproportionately expensive. Second, and just as important in a cloud-native world: vertical scaling isn't elastic. Resizing a running machine's CPU/RAM allocation typically requires real downtime or at least a restart — it's not something you do smoothly, automatically, in direct response to a traffic spike that might last five minutes and then vanish.
Horizontal scaling sidesteps both limits. There's no hard ceiling — you're not waiting for a bigger machine to exist, you're adding another ordinary-sized one, and you can generally keep doing that far past where vertical scaling would have stalled. And critically, it's elastic: starting a new container instance takes seconds, not a maintenance window, so the number of running instances can track real, moment-to-moment demand — scaling out when traffic spikes, and scaling back in once it subsides, without ever touching a machine that's already serving traffic. This is precisely why horizontal scaling is the dominant, preferred approach in cloud-native architecture, and why containers (covered earlier in this Part) are such a natural fit for it — a container image is designed to be started as many identical, disposable copies as needed.
Both matter, and they're not competing — a well-scaled cloud-native system does both: each individual instance is written to use its own resources efficiently (Part IV), and the system as a whole runs many of those efficient instances side by side (this lesson).
Horizontal scaling only behaves correctly under one real, non-negotiable condition: the application must be stateless. A stateless instance keeps no data in memory that only it knows about between requests — every request should be servable correctly by any instance, not specifically the one that handled the user's previous request.
IMemoryCache (lesson 265) — that only Instance A knows about.This is exactly the problem lesson 265 already flagged directly: IMemoryCache is a fast, in-process cache scoped to one running instance, and "the moment your app scales to more than one instance behind a load balancer, each instance's IMemoryCache becomes its own island." The fix isn't to work around horizontal scaling — it's to externalize the shared state so every instance reads from and writes to the same place:
IMemoryCache entries only that instance knows aboutIDistributedCache (lesson 266) — one shared cache every instance reads and writesThe exact same product-lookup method, written two ways — one that quietly breaks under horizontal scaling, and one that doesn't:
// Breaks the moment you run more than one instance
public class ProductService(AppDbContext db, IMemoryCache cache)
{
public async Task<Product?> GetAsync(int id) =>
await cache.GetOrCreateAsync($"product:{id}", async entry =>
{
entry.SlidingExpiration = TimeSpan.FromMinutes(10);
return await db.Products.FindAsync(id);
});
// Each instance builds its own separate copy of this cache.
// A write that invalidates it on Instance A never reaches Instance B.
}
// Safe under any number of instances
public class ProductService(AppDbContext db, IDistributedCache cache)
{
public async Task<Product?> GetAsync(int id)
{
var cached = await cache.GetStringAsync($"product:{id}");
if (cached is not null)
return JsonSerializer.Deserialize<Product>(cached);
var product = await db.Products.FindAsync(id);
if (product is not null)
await cache.SetStringAsync($"product:{id}",
JsonSerializer.Serialize(product),
new DistributedCacheEntryOptions { SlidingExpiration = TimeSpan.FromMinutes(10) });
return product;
}
// Every instance reads and writes the same shared store (lesson 266) —
// it doesn't matter which instance served the previous request.
}An order/payment API normally runs as 3 instances, handling routine traffic comfortably. A flash sale is announced, and request volume climbs to 15x normal within minutes. Because the service is genuinely stateless — order data lives in the shared database, cart/session data lives in a distributed cache (266), nothing important lives only in one instance's process memory — the platform's auto-scaler can respond by simply starting more identical instances, and the load balancer immediately begins routing traffic across all of them. No instance needs to "know" it's now one of 20 instead of one of 3; each one just keeps handling whichever requests the load balancer sends it, exactly as before.
When the sale ends and traffic drops back to normal, the same mechanism scales the instance count back down — again, safely, precisely because no instance was the sole holder of anything another instance, or a future instance, might need.
Horizontal scaling's elasticity is usually driven automatically, not by a human watching a dashboard and manually starting instances. An auto-scaler continuously watches one or more signals and adjusts the running instance count to match:
| Trigger | What it measures |
|---|---|
| CPU utilization | Average CPU usage across running instances — scale out when consistently high, scale in when consistently low |
| Memory utilization | Similar idea, applied to memory pressure instead of CPU |
| Request queue depth | How many requests are waiting to be handled — a direct signal that current capacity isn't keeping up |
| Custom application metrics | A domain-specific signal your app itself exposes — e.g. "pending orders in the processing queue" — driving scale decisions tailored to what actually matters for that specific workload |
Whichever signal drives it, the mechanism is the same at its core: the auto-scaler compares the current value against a configured threshold, and adds or removes running instances to bring the system back toward the target. This is only meaningfully safe to do automatically, without human oversight of every scaling event, because a stateless instance can be added or removed at any moment without anyone needing to migrate state off it first.
Vertical scaling is like replacing a bank's one teller with a single, extraordinarily fast super-teller who can process transactions quicker than any normal teller. It helps — until you hit the physical limit of how fast one person can possibly work, no matter how skilled.
Horizontal scaling is like opening more teller windows, each staffed by an ordinary, interchangeable teller, with a line-management system directing each customer to whichever window is free — that's the load balancer. This scales much further, and you can open or close windows as the line gets longer or shorter throughout the day.
But it only works smoothly if every teller has access to the same shared record of each customer's account — the bank's central ledger, not a private notebook one specific teller keeps in their own drawer. If Teller 3 wrote your balance update in their own private notebook, and your next visit gets routed to Teller 7, Teller 7 has no idea what happened. That private notebook is exactly what in-process state and IMemoryCache become once you have more than one instance — the shared ledger is exactly what IDistributedCache and a shared database provide instead.
This is the connection worth internalizing above everything else in this lesson. Part IX taught you idempotency (290), retries (291), and eventual consistency (295) as patterns for building resilient distributed systems. Once your application runs as more than one instance, those patterns stop being "distributed systems best practice you might reach for eventually" and become facts about how your own, single application already behaves in production, every day.
None of this is a single-instance concern. A single instance handling one request at a time never has to worry about "what if a different copy of me is handling a retry of this same logical operation right now." The moment there's more than one instance, that scenario isn't a rare edge case anymore — it's a routine, expected possibility every single day the system runs under real, retried, occasionally-duplicated traffic.
It's tempting to think horizontal scaling is entirely something the platform team configures, with no impact on application code. Statelessness proves that wrong: whether your app is safe to horizontally scale is decided by how the application itself is written — where it keeps state — not by any infrastructure setting alone.
Making one instance efficient (209, 221) and running many instances (this lesson) solve different problems and both matter. An inefficient instance, horizontally scaled, just means many inefficient instances running side by side — you generally want both: efficient instances, run in the right quantity for current demand.
Defaulting to vertical scaling whenever load grows, without considering elasticity — a bigger machine has to be provisioned in advance and generally can't shrink back down smoothly once the spike passes.
Default to horizontal scaling for anything with variable, unpredictable load — it's what actually tracks demand up and down automatically.
Building request handling that implicitly assumes "if this fails and gets retried, it'll be the same instance handling it again" — a load balancer gives you no such guarantee.
Design every operation to be safely idempotent (290), regardless of which instance ends up handling any given attempt.
Not revisiting caching choices when moving from one instance to many — IMemoryCache doesn't fail loudly, it just quietly produces inconsistent results per instance.
Move genuinely shared cache data to IDistributedCache (266) the moment more than one instance is in play.
IMemoryCache (265) breaks the moment a load balancer routes a user's next request to a different instance. Use IDistributedCache (266) and the strategies from lesson 280 instead.You've seen why running many instances is a genuinely different challenge from making one instance fast. Let's confirm the reasoning holds up.
1. Why is horizontal scaling generally preferred over vertical scaling for a cloud-native service with unpredictable, spiky traffic?
Correct: B
Why B is correct: Elasticity and the absence of a single-machine ceiling are exactly why horizontal scaling dominates cloud-native design — it can respond to a spike in minutes and scale back down just as easily.
Why A is incorrect: The opposite is true — horizontal scaling specifically requires the application to be stateless to work correctly; that's the core prerequisite this lesson covers.
Why C is incorrect: Vertical scaling remains fully supported; it's just a poorer fit for elastic, unpredictable load.
Why D is incorrect: Horizontal scaling specifically requires a load balancer to distribute traffic across the running instances — it doesn't eliminate the need for one.
Reinforcement: Elasticity and no hard ceiling are the concrete, technical reasons horizontal scaling wins for variable load — not vague "it's more modern" reasoning.
2. A service stores each logged-in user's shopping cart in an IMemoryCache entry. After the service is horizontally scaled to 4 instances behind a load balancer, users report their carts randomly appear empty. What's the root cause?
Correct: B
Why B is correct: This is exactly the failure mode lesson 265 warned about and this lesson builds on — IMemoryCache is per-instance. A request landing on a different instance than the one that stored the cart simply won't find it there.
Why A is incorrect: There's no such cross-instance limit; the real issue is each instance has an entirely separate cache, not a shared one with a capacity problem.
Why C is incorrect: Load balancers route traffic; they don't inspect or delete application-level cache data.
Why D is incorrect: Scaling doesn't clear existing instances' caches — the problem is that each instance never had the other instances' data in the first place.
Reinforcement: The fix is IDistributedCache (266) — one shared store every instance reads and writes, so it doesn't matter which instance serves any given request.
3. Why do idempotency and retries (Part IX) become necessary, not just theoretically useful, once an application runs as multiple horizontally-scaled instances?
Correct: B
Why B is correct: This is the lesson's central connection back to Part IX — once more than one instance exists, a retried request landing on a different, equally healthy instance is normal, everyday behavior, and idempotency is what keeps that from causing duplicate side effects like a double charge.
Why A is incorrect: Horizontal scaling is about capacity and elasticity, not per-request speed, and doesn't inherently slow anything down.
Why C is incorrect: Exception handling is unrelated to and unaffected by how many instances are running.
Why D is incorrect: Load balancers forward retried requests exactly like any other request — they have no special restriction against retries.
Reinforcement: More instances means "the same logical operation, handled independently more than once" stops being unlikely and starts being routine — design for it accordingly.
4. Which auto-scaling trigger would be most appropriate for a service where slow requests are best predicted by how many requests are currently waiting to be processed, rather than by raw CPU usage?
Correct: B
Why B is correct: Request queue depth directly measures the exact symptom described — work piling up faster than it's being processed — which can be a more direct, earlier signal than CPU usage for I/O-bound or unevenly-loaded workloads.
Why A is incorrect: CPU utilization is a valid trigger in general, but the scenario specifically describes a case where it's a poorer fit than queue depth.
Why C and D are incorrect: Neither reflects real-time load or demand in any way — they're static, build-time properties with no bearing on current traffic.
Reinforcement: Different workloads are better served by different auto-scaling signals — CPU/memory, queue depth, or a custom application-defined metric — chosen based on what actually predicts the service falling behind.
You now understand what actually changes when an application moves from "efficient on one machine" to "running as many instances at once" — and why statelessness and the Part IX resilience patterns are what make that shift safe.
dotnetmadeeasy.com — Learn C# and .NET, the right way.