Scalability Foundations — From One Box to a Million QPSScalability Fundamentals: Vertical vs Horizontal, Load Balancing, Statelessness

The Problem

Day one of your product: 1k DAU, one EC2 box running Postgres and Node, everything fine. Six months later you're at 1M DAU and 3k QPS, p99 latency has gone from 80ms to 2.4s, and the DB drops out occasionally. This story is almost universal — Instagram carried 14M users with three engineers in 2010; Notion moved from a single Postgres instance to a sharded fleet around 2019. Same arc, every time.

This issue isn't "throw more boxes at it." It's about figuring out which loads should be scaled up vs out, when you actually have to go horizontal, what hidden costs come with that, and why stateless services are the price of admission for horizontal scale.

Requirements and Constraints (the interview opener)

• Read/write ratio: 100:1 (social feeds) or 1:1 (instant messaging)? Determines whether you scale reads with replicas first, or shard writes.
• QPS magnitude: 100, 10k, or 1M? Below 10k QPS you almost never need distributed anything; at 1M you're forced into sharding and CDN.
• Data volume and growth curve: 1TB or 1PB? A single Postgres instance is comfortable up to about 10TB in practice.
• Latency SLO: p99 at 100ms (search) or 10ms (ad bidding)? The latter rules out any cross-region hop.
• Consistency requirements: bank-transfer strong, or like-button eventual?
• Budget: an m5.large is a few tens of dollars a month; a c7gn.16xlarge is thousands. Often overlooked as a constraint.

High-Level Architecture (one box to N tiers)

graph TD
    C[Client]
    DNS["DNS / GeoDNS"]
    L4["L4 LB · LVS
~1M QPS ingress"]
    L7["L7 LB · Nginx/Envoy
routing · TLS · rate limit"]
    A1["App-1
stateless"]
    A2["App-2
stateless"]
    AN["App-N
stateless"]
    R[("Redis
Cluster")]
    PG[("PG Primary")]
    R1[("Replica-1
read replica")]
    R2[("Replica-2
read replica")]

    C --> DNS --> L4 --> L7
    L7 --> A1 & A2 & AN
    A1 & A2 & AN --> R
    A1 & A2 & AN --> PG
    PG -.replicate.-> R1
    PG -.replicate.-> R2

    classDef edge fill:#1a2530,stroke:#64c8ff,color:#e8eef5
    classDef app fill:#0e2030,stroke:#5eead4,color:#e8eef5
    classDef data fill:#1a1a30,stroke:#ffb450,color:#e8eef5
    class C,DNS,L4,L7 edge
    class A1,A2,AN app
    class R,PG,R1,R2 data

Key Technical Points

1. Vertical vs Horizontal Scaling

Principle: vertical scaling = bigger machine (more CPU / RAM / NVMe); horizontal scaling = more machines. Vertical scaling stays roughly linear in price-performance until you hit x86 memory bandwidth or NUMA limits. A single box today can reach 192 vCPU / 2TB RAM (c7g.metal, x2idn.32xlarge) — so don't go distributed too early.

2. Load Balancer: L4 vs L7

Principle: L4 (TCP/UDP) only inspects the four-tuple, forwards without unpacking, and can do 2M+ QPS per box (LVS/IPVS, AWS NLB). L7 (HTTP) can route by path, header, or cookie and handle TLS termination, compression, and rate limiting — but the CPU cost is much higher (Envoy does roughly 50k QPS per core). In production you typically see L4 in front, L7 behind.

# P2C pseudocode (Envoy default)
def pick_backend(backends):
    a, b = random.sample(backends, 2)
    return a if a.active_requests <= b.active_requests else b
# Math: with N backends, peak load drops from O(log N) to O(log log N)

3. Stateless Services (the ticket to horizontal scale)

Principle: service instances hold no "state needed for the next request" — sessions, caches, counters all live externally in Redis or a DB. Then the LB can route any request to any instance, and a crash hurts no one.

# Anti-pattern: sessions in memory
sessions = {}  # ❌ lost on restart, scale-up drops users, A's session invisible to B

# Correct: JWT (stateless) or Redis (shared state)
def auth(req):
    token = req.headers["Authorization"]
    return jwt.verify(token, PUBLIC_KEY)  # zero server-side state

4. Capacity Planning and "Good Enough"

Principle: don't reach for Kubernetes and microservices on day one. Shopify handled 76M peak QPS on Black Friday with a monolithic Rails app on a large pod fleet. Stay monolithic as long as you can; add machines before sharding. Back-of-envelope math is core interview material:

# Example: designing a Twitter timeline
# 300M DAU × 100 refreshes/day = 3e10 reads/day
# = 3e10 / 86400 ≈ 350k QPS average, peak ~3× = 1M QPS
# A 4-core box handles ~5k QPS of static reads
# → at least 200 read instances (with 2× headroom)

Scaling Up: Where the Next Bottlenecks Live

1. The first bottleneck is usually DB writes: reads relax with replicas; writes either scale vertically or shard. When you see p99 write latency > 100ms, it's time.
2. The second is cache hot keys: one popular key saturates a Redis shard. You need a local cache plus multiple replicas.
3. The third is the network: cross-AZ traffic is billed per GB (~$0.01/GB on AWS); too-fine microservice slicing gets schooled by transfer fees and RTT.
4. Multi-region: when users span more than two continents, you introduce GeoDNS, in-region closure, and async global replication.

Common Pitfalls

• Going distributed too early: 1k QPS on Kubernetes with eight microservices, and debugging takes 10× longer than the monolith would have.
• The sticky-session trap: binding instances by cookie hash means scale-up or restart drops many users, and a hot instance has no relief valve.
• LB with no health checks: a half-dead instance (accepts connections, doesn't respond) still gets traffic, and every RTT spikes. Always configure active health checks + outlier ejection (Envoy's P2C also auto-ejects).
• No capacity drills: not load-testing before Black Friday, then learning your connection pool ceiling during the actual event. Netflix's Chaos Monkey culture exists for this reason.
• Ignoring the thundering herd: when a cache expires, 1000 instances hit the DB at once. Solve with singleflight, mutexes, or probabilistic early refresh.

Sample Interview Questions

1. "You're at 5k QPS p99 200ms; you need 50k QPS p99 100ms. What's your five-step plan?" (Expect to ask about read/write ratio and current bottleneck before proposing solutions.)
2. "Why not just buy one 192-core box?" (Tests your grasp of vertical scaling ceilings, blast radius, and SPOF.)
3. "Where do L4 and L7 LBs go, and why not just use L7 everywhere?"
4. "If you move sessions to stateless, do you pick JWT or Redis sessions?" (Tests trade-offs around revocation, payload size, security.)
5. "How does round robin fall apart with long-lived connections? How do you fix it?" (Leads into least-conn / P2C.)

Key Resources

• 📕 Designing Data-Intensive Applications (Kleppmann), Ch. 1: Reliable, Scalable, and Maintainable Applications.
• 📝 Stack Overflow Architecture (2016, Nick Craver) — the canonical case for vertical scaling to its limit. nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/
• 📝 High Scalability blog: the classic "Power of Two Random Choices" article.

English Summary

Scaling means surviving load growth without rewriting from scratch. Vertical scaling (bigger box) is underrated — Stack Overflow runs billions of pageviews on a handful of fat SQL servers. Horizontal scaling demands stateless services, smart load balancing (L4 for throughput, L7 for routing; P2C beats round-robin under skewed load), and capacity planning via back-of-envelope math. The cardinal sin is premature distribution: complexity grows superlinearly, debuggability collapses. Scale when you must, not when you can.

Going Deeper (click to expand)