Real-time Systems — Keeping the World Consistent Within 100msTransport, Game Netcode, Authoritative State, Collaborative Sync

Problem Scenario + Constraints

Design the backend for a real-time multiplayer battle with 100k concurrent players (.io-style / lightweight FPS): players in the same room move and shoot, and everyone must see everyone else's actions "almost instantly." This is a different species from a request-response web backend. Its SLO isn't "p99 < 200ms" — it's end-to-end p95 < 100ms with bounded jitter, because the human eye is extremely sensitive to input lag above ~50ms and to position teleports.

• Latency SLO: local input response < 16ms (one frame); a peer's action visible < 100ms. Jitter is deadlier than absolute latency.
• Update rate: server tick 20–60Hz, each room broadcasts state 20–60 times/sec; 20–100 players per room.
• Scale: 100k players ÷ 50 per room = 2000 concurrent rooms; 30Hz × 50 players ≈ 1500 outbound packets/sec/room, millions of pps globally.
• Consistency: games need an authoritative server (anti-cheat); collaborative editing needs eventual consistency (never drop a keystroke). The two flavors of real-time have completely different consistency models.
• Reliability: dropping one position packet is fine (next tick overwrites it); dropping a "fire" event is an incident — reliable and unreliable channels must be separated.

High-Level Architecture

graph TD
    C["Client
local prediction + render interp."]
    MM["Matchmaker
HTTPS · assign room/region"]
    GW["Edge Gateway
terminate UDP/WebRTC · nearest PoP"]
    GS["Authoritative Game Server
room process · tick loop"]
    R[("Redis
session/room state")]
    DB[("Persistent DB
results/leaderboard")]

    C -->|① find match HTTPS| MM
    MM -->|② return server+token| C
    C <-->|③ realtime bidir UDP| GW
    GW <-->|④ forward| GS
    GS -.room state.-> R
    GS -.async persist.-> DB

    classDef client fill:#1a2530,stroke:#64c8ff,color:#e8eef5
    classDef edge fill:#0e2030,stroke:#5eead4,color:#e8eef5
    classDef core fill:#1a1a30,stroke:#ffb450,color:#e8eef5
    classDef store fill:#2a1530,stroke:#ff7ab6,color:#e8eef5
    class C client
    class MM,GW edge
    class GS core
    class R,DB store

Control plane (matchmaking, login) goes over HTTPS; the data plane (real-time state) uses a dedicated low-latency channel — the two paths are decoupled

Responsibilities: the Matchmaker is a plain stateless web service that assigns players to rooms in the nearest region by geography and skill, returning a server address + auth token. The Edge Gateway terminates UDP/WebRTC within ~30ms of the user, handles NAT traversal and DDoS protection. The Game Server is a stateful room process running a fixed-rate tick loop: collect all player inputs this tick → advance physics → broadcast a new snapshot. State lives in memory; Redis only does crash recovery and cross-process coordination; the DB asynchronously records results. The key is to fully separate the "real-time data plane" from the "durable control plane."

Key Technical Points

1. Transport Choice: Why Real-time Systems Flee TCP

Core trade-off: TCP's reliable ordering is precisely the poison for real-time systems.

Principle: TCP guarantees the byte stream arrives in order — which means once a packet is lost, packets that have already arrived must queue in the kernel buffer waiting for retransmission; the application can't read them. This is Head-of-Line (HoL) blocking. In a game, if position packet #100 is lost, #101 and #102 have arrived but are stuck; by the time #100 is retransmitted, they are already stale — you paid an RTT of stutter for data nobody wanted. A real-time system would rather drop stale data and move on than stay in order. So native games use UDP (implementing selective reliability on top), browsers use WebRTC DataChannel (UDP-based, configurable unreliable/unordered) or the newer WebTransport (QUIC) — QUIC implements multiple streams in user space, so loss on one stream doesn't block others, sidestepping kernel TCP's HoL blocking.

The practical approach is dual channels: position/orientation over unreliable UDP (drop = no problem, next tick overwrites); "fire/pickup/death" critical events over a reliable channel (custom ACK retransmit, or WebRTC's reliable mode).

2. Client-Side Prediction + Server Reconciliation: Hiding the Speed of Light

Core trade-off: trade optimistic local simulation (that may need rollback) for zero-perceived-latency feel.

Principle: even at 50ms RTT, "press → send to server → wait for reply → then move" feels sluggish. The fix is a trio: ① Client-side prediction — move locally immediately on input without waiting for the server, while sending the sequence-numbered input. ② Server reconciliation — the server is authoritative; its reply carries "processed up to input #N + authoritative position." The client takes that position as the baseline and replays all local inputs after #N that haven't yet been acknowledged, arriving at a corrected current position. If the prediction was right, the screen doesn't move; if interrupted (e.g. hit a wall), it corrects smoothly. ③ Entity interpolation — other players' states arrive discretely per tick; the client deliberately renders ~100ms behind, interpolating between two known snapshots for smoothness instead of teleporting. The cost: the "others" you see are always 100ms in the past — which is what lag compensation solves: when resolving a shot, the server rewinds the target to where the shooter actually saw it before computing the hit.

# Client prediction + reconciliation (pseudocode)
pending = []                      # unacknowledged local inputs
def on_input(cmd):
    cmd.seq = next_seq()
    apply_local(cmd)              # ① move locally now, zero-latency feel
    pending.append(cmd)
    net.send(cmd)                 # send seq-numbered input to authoritative server

def on_server_snapshot(snap):
    me.pos = snap.pos             # ② trust authoritative position
    # drop inputs already confirmed by the server
    pending = [c for c in pending if c.seq > snap.last_processed_seq]
    for c in pending:             # replay still-unconfirmed inputs
        apply_local(c)            # → right prediction: no jump; wrong: smooth correction

# other players: buffer two frames, interpolate at render_time = now - 100ms (③)

3. Authoritative Server & State Sync: Ticks, Snapshots, Interest Management

Core trade-off: sync fidelity vs bandwidth/CPU — sending "the whole world" to everyone every tick gets killed by both bandwidth and CPU.

Principle: the authoritative server runs a fixed-rate tick loop (e.g. 30Hz): absorb all inputs this tick → advance deterministic physics → produce new world state → broadcast. The broadcast has three layers of optimization: ① Delta compression — send only changes relative to the last snapshot this client acknowledged; static objects cost no bandwidth. ② Interest management (AOI, Area of Interest) — players only need nearby entities; using a grid/quadtree spatial index, each player receives only updates within their view. A 100-player room broadcasting to all is O(n²)=10k messages/tick; with AOI each player sees only ~10 nearby, dropping to O(n·k). ③ Priority/rate tiering — nearby entities update at high frequency, distant ones lower or not at all. This turns "full state every tick" from a bandwidth disaster into an engineerable, stable stream.

4. Collaborative Sync: The Other Real-time — Eventual, Not Authoritative

Core trade-off: OT's centralized simplicity vs CRDT's decentralized freedom.

Principle: games want "authoritative + anti-cheat," but Figma/Notion/Google Docs collaborative editing is a different real-time: it must work offline, allow concurrent edits to the same document, never drop a keystroke, and eventually converge everyone to the same state. Two schools: OT (Operational Transformation) — transform concurrent operations against each other to preserve intent (you insert at position 5, I delete at position 3, OT adjusts your index); relies heavily on a central server for ordering, complex logic but compact state (Google Docs). CRDT (Conflict-free Replicated Data Type) — design the data so it is mathematically commutative and converges regardless of apply order, naturally supporting P2P and offline, but with large metadata overhead (each character carries a unique ID and tombstone). LWW (Last-Write-Wins) is the simplest CRDT form: each field independent, the last writer wins.

# LWW register (simplest CRDT): a total order on (timestamp, replica_id) decides
def merge(local, remote):
    if (remote.ts, remote.node) > (local.ts, local.node):
        return remote          # "newer" write wins; merge is commutative/idempotent
    return local
# The hard part isn't merge, it's the GRANULARITY of LWW:
# whole object? edits overwrite each other and get lost;
# per-attribute? concurrent edits to different attrs never conflict — Figma chose the latter

Scaling & Optimization

• Room sharding & nearest access: route players to the geographically nearest region (GeoDNS / Anycast ingress); cross-region matches use relays. A room is a natural shard unit — a single room can't be split, so scale horizontally by adding machines for more rooms.
• Elasticity of stateful services: game servers are stateful; you can't kill processes freely like stateless web. Use draining: stop assigning new rooms and let existing matches finish before retiring.
• Crash recovery: when a room process crashes, all in-memory state is lost. Periodically checkpoint critical state (score, key events) to Redis to rebuild an approximate state; pure position state is acceptable to lose.
• Anti-cheat: never trust the client. Validate movement server-side (speed caps, wall-clip detection); hit detection is server-authoritative, the client only "renders + predicts."
• Bandwidth keeps growing: beyond delta, add quantized compression (positions as fixed-point not float), bit-packing, distance-based precision reduction.

Common Pitfalls + Interview Follow-ups

Frequent interview questions: ① Why don't games use TCP? How exactly does HoL blocking hurt? ② How do you correct a wrong prediction without it looking jarring? ③ How do you keep bandwidth from exploding with 100 players per room per tick? ④ Why must hit detection be server-authoritative, and how does lag compensation stay fair? ⑤ How do the consistency models of Figma collaboration and an FPS match differ, and why?

Deeper Resources

• Gabriel Gambetta — Fast-Paced Multiplayer series: prediction/reconciliation/interpolation/lag compensation in four steps, with a runnable demo (gabrielgambetta.com).
• Valve Developer Community — Source Multiplayer Networking & Lag Compensation: first-hand docs on industrial-grade authoritative-server netcode.
• Discord Engineering — Handling 2.5M Concurrent Voice Users with WebRTC: SFU, regionalization, silence optimization, large-scale real-time A/V experience.
• Figma Blog — How Figma's multiplayer technology works: LWW register, tree consistency, why not a true CRDT.
• Designing Data-Intensive Applications (Kleppmann), Ch. 5: replication and concurrent write conflicts — the foundation for understanding LWW/CRDT convergence.

Deeper Thinking (click to expand)