Scalability
Scalability is the ability of a system to handle growth — more users, more data, more traffic — without a proportional loss in performance or availability. It is not a single technique but a set of architectural decisions that compound over time.
There are two primary dimensions of scale. Scale up means making individual machines bigger and faster. Scale out means adding more machines and distributing work among them. Modern distributed systems overwhelmingly prefer scale out because commodity hardware is cheaper, fault tolerance is easier, and the ceiling is effectively unlimited.
This guide covers the building blocks of scalable systems: load balancing, stateless architectures, caching, database scaling, and auto-scaling. Each topic is accompanied by practical patterns and the tradeoffs they introduce.
Vertical scaling, or scaling up, increases the resources of a single node: more CPU cores, more RAM, faster disks, better network cards. It is simple because the architecture does not change, but it has hard physical and economic limits. The largest single machine is always smaller than the largest cluster.
Horizontal scaling, or scaling out, adds more nodes to a pool and distributes work across them. It introduces complexity — coordination, load balancing, data partitioning — but it also provides elasticity, redundancy, and the ability to use commodity hardware.
| Dimension | Vertical Scaling | Horizontal Scaling |
|---|---|---|
| Cost curve | Exponential for high-end hardware | Roughly linear per unit of capacity |
| Fault tolerance | Single point of failure | Resilient to individual node failures |
| Complexity | Low — no distributed-system concerns | High — networking, consensus, partitioning |
| Elasticity | Slow — hardware procurement or VM resize | Fast — spin up containers or instances |
| Ceiling | Limited by largest available machine | Effectively unbounded |
best practice
A load balancer distributes incoming traffic across a pool of backend servers. It is the simplest way to add horizontal scale and fault tolerance to the application tier. Without it, a single popular server becomes a hotspot and a single point of failure.
| 1 | ┌─────────────┐ |
| 2 | │ Client │ |
| 3 | └──────┬──────┘ |
| 4 | │ |
| 5 | ┌──────▼──────┐ |
| 6 | │ DNS / │ |
| 7 | │ CDN │ |
| 8 | └──────┬──────┘ |
| 9 | │ |
| 10 | ┌──────▼──────────────────────────────┐ |
| 11 | │ Load Balancer (L4/L7) │ |
| 12 | │ Round-Robin / Least-Connections / │ |
| 13 | │ IP Hash / Consistent Hash │ |
| 14 | └───┬───────┬───────┬───────┬─────────┘ |
| 15 | │ │ │ │ |
| 16 | ┌───▼───┐ ┌─▼─────┐ ┌─▼─────┐ ┌─▼─────┐ |
| 17 | │ App-1 │ │ App-2 │ │ App-3 │ │ App-4 │ |
| 18 | └───────┘ └───────┘ └───────┘ └───────┘ |
Layer 4 vs Layer 7
Layer 4 load balancers operate on transport-layer information such as IP address and TCP port. They are fast and stateless but cannot inspect HTTP headers or route based on URL paths. Layer 7 load balancers understand application protocols, enabling path-based routing, SSL termination, sticky sessions, and request rewriting.
| Algorithm | How It Works | Best For |
|---|---|---|
| Round-Robin | Cycles through backends in order | Homogeneous servers, equal load |
| Least-Connections | Routes to the backend with fewest active connections | Variable request durations |
| IP Hash | Hashes client IP to pick a backend | Session affinity without cookies |
| Consistent Hash | Hashes a key and maps it to a ring | Cache servers, minimizing reshuffling |
| Weighted | Assigns proportionally more traffic to powerful nodes | Heterogeneous fleet |
warning
A stateless service does not store client-specific data between requests. Any request can be handled by any instance, which makes horizontal scaling trivial. A stateful service retains information about the client or connection, such as a WebSocket session or an in-memory shopping cart.
Stateless does not mean the system has no state. It means the state lives outside the application server — in a database, cache, object store, or message queue. This separation is the single most important enabler of elastic scale.
| 1 | Stateless request flow: |
| 2 | |
| 3 | Client ──▶ LB ──▶ App Server A |
| 4 | reads/writes state from Redis/DB |
| 5 | returns response |
| 6 | |
| 7 | Next request: |
| 8 | Client ──▶ LB ──▶ App Server B |
| 9 | reads same state from Redis/DB |
| 10 | no loss of context |
| Aspect | Stateless | Stateful |
|---|---|---|
| Scalability | Trivial — add nodes behind LB | Hard — must route users to right node |
| Failover | Instant — any node can take over | Complex — state must be migrated or rebuilt |
| Use cases | REST APIs, microservices, batch workers | WebSockets, game servers, real-time collab |
info
Caching is the fastest way to reduce load on databases and improve response times. A cache stores copies of frequently accessed data in fast memory, closer to the consumer. The cache hit rate determines the effectiveness of the strategy.
Caches exist at every layer of the stack: browser caches for static assets, CDN edge caches for content, application caches for hot objects, and database buffer pools for frequently read pages. The closer the cache is to the user, the lower the latency and the less load on origin systems.
| 1 | Read path with cache-aside: |
| 2 | |
| 3 | 1. App receives request for user profile |
| 4 | 2. Check Redis cache for key user:1234 |
| 5 | 3. If cache HIT: return value immediately |
| 6 | 4. If cache MISS: |
| 7 | a. Read from database |
| 8 | b. Write result to cache with TTL |
| 9 | c. Return result |
| 10 | |
| 11 | Write path: |
| 12 | 1. Update database |
| 13 | 2. Invalidate cache key (or update it) |
| 14 | 3. Next read repopulates cache |
best practice
Databases are often the first bottleneck in a growing system. Scaling them involves a progression of techniques: vertical scaling, read replicas, caching, partitioning, sharding, and eventually specialized databases for different access patterns.
Read Replicas
Read replicas copy the primary database's data asynchronously. They offload read traffic and improve availability. The tradeoff is replication lag: a write on the primary may not be visible on replicas for milliseconds or seconds. Applications must tolerate stale reads or route critical reads to the primary.
Sharding
Sharding splits data across multiple database instances based on a shard key. Each shard holds a subset of the data, allowing the system to scale write throughput linearly. The challenge is choosing a shard key that distributes load evenly and minimizing cross-shard queries.
| 1 | Shard key selection for a user database: |
| 2 | |
| 3 | Option 1: user_id % number_of_shards |
| 4 | - Even distribution if user_id is random |
| 5 | - Hot shard risk if a region or tenant dominates |
| 6 | |
| 7 | Option 2: Hash(user_id) |
| 8 | - Better distribution than modulo |
| 9 | - Resharding requires remapping all data |
| 10 | |
| 11 | Option 3: Range-based (user_id 1-1M → shard 1) |
| 12 | - Easy range queries within a shard |
| 13 | - Risk of hot ranges (new signups hit latest shard) |
| 14 | |
| 15 | Option 4: Composite (tenant_id + user_id) |
| 16 | - Groups a tenant's data together |
| 17 | - Can create tenant-level hot shards |
| Technique | Scales | Complexity | Tradeoff |
|---|---|---|---|
| Vertical scaling | Single node | Low | Hard ceiling, single point of failure |
| Read replicas | Read throughput | Low | Replication lag, no write scaling |
| Caching | Hot reads | Medium | Stale data, invalidation complexity |
| Partitioning | Single DB, large tables | Medium | Query planner complexity |
| Sharding | Write throughput | High | Cross-shard queries, resharding pain |
warning
Auto-scaling adjusts the number of running instances based on demand. It prevents over-provisioning during quiet periods and under-provisioning during spikes. Cloud providers offer horizontal pod autoscalers, VM scale sets, and serverless functions that scale to zero.
| 1 | Common auto-scaling metrics: |
| 2 | |
| 3 | - CPU utilization → scale when compute-bound |
| 4 | - Request latency (p95/p99) → scale when response times degrade |
| 5 | - Request queue depth → scale when backlog grows |
| 6 | - Custom metrics → business events, queue length, error rate |
| 7 | |
| 8 | Scaling policies: |
| 9 | - Target tracking → keep metric near a setpoint |
| 10 | - Step scaling → add/remove fixed capacity at thresholds |
| 11 | - Scheduled scaling → pre-warm before known events |
| 12 | - Predictive scaling → ML-based forecast of upcoming load |
Auto-scaling is not instantaneous. It takes time to launch instances, pull container images, and pass health checks. If your traffic spikes faster than your scale-up time, you need a combination of caching, rate limiting, and over-provisioned headroom.
info
Beyond the basics, several architectural patterns help systems scale under specific constraints. Knowing when to apply them separates mid-level from senior engineers.
CQRS
Command Query Responsibility Segregation separates read and write models. Writes go through a strict, normalized model; reads are served from denormalized, query-optimized views. This pattern shines when read and write patterns differ dramatically.
Event Sourcing
Instead of storing only the current state, event sourcing stores a log of immutable events. State is derived by replaying events. It provides a complete audit trail and enables temporal queries, but it adds complexity to schema evolution and snapshotting.
Bulkheads
Bulkheads isolate failures by partitioning resources. If one customer or feature exhausts a pool, the rest of the system continues to function. This is a reliability pattern that also improves scalability under stress.
Rate Limiting
Rate limiting prevents any single client from overwhelming the system. It protects downstream services and ensures fair resource allocation. Token bucket and sliding window are the most common algorithms.
pro tip