Reliability — One flaky dependency shouldn't take down the whole chainCircuit Breaker, Retry/Backoff, Bulkhead & Graceful Degradation

Problem & Constraints

Design an e-commerce checkout path: 50K orders/sec, p99 SLO 300ms, overall availability target 99.95% (~4.4h downtime/year). Placing an order synchronously calls roughly 30 downstream dependencies — inventory, coupons, fraud, address, payment pre-auth… Only inventory and payment are critical; the rest are non-critical.

What you really need to prevent isn't "a service died" but cascading failure: one GC pause in the coupon service pushes its response from 20ms to 5s. The caller waits synchronously, every order request pins a thread blocking on coupons — within seconds the thread pool is exhausted and even requests that never touch coupons can't get in. One non-critical dependency drags down the entire checkout. The core thesis of reliability engineering: keep local failures local. This issue covers four complementary weapons: circuit breaker, retry with backoff, bulkhead, graceful degradation.

• Failure model: downstreams rarely die outright — they go slow. Slow is worse than dead, because it quietly drains your threads/connections.
• Dependency tiering: critical vs non-critical. Non-critical deps must be skippable, or their availability becomes your floor.
• Serial availability: 30 deps in series, each 99.9% → overall ≈ 99.9%³⁰ ≈ 97%. Without isolation, more deps = more fragile.
• Idempotency: determines whether you can safely retry (see Day 7, Day 17).

High-Level Architecture

Every downstream call passes through a resilience layer: timeout → bulkhead (isolated resource pool) → circuit breaker (tracks error rate) → retry (with backoff). If any stage decides failure, it immediately goes to a fallback (cached value / default / skip) instead of blocking. At the ingress there's also load shedding, which under overload protects critical requests first.

graph LR
    IN["order request"]
    LS{"load shed
overload→drop"}
    ORD["order svc
orchestrate"]
    BH["bulkhead
pool per dep"]
    CB{"breaker
open?"}
    DEP["downstream
coupon/fraud..."]
    FB["fallback
cache/default/skip"]

    IN --> LS -->|admit| ORD --> BH --> CB
    LS -.->|reject 429| IN
    CB -->|closed| DEP
    CB -.->|open→fail fast| FB
    DEP -.->|timeout/error| FB

    classDef in fill:#1a2530,stroke:#64c8ff,color:#e8eef5
    classDef gate fill:#2a1530,stroke:#ff7ab6,color:#e8eef5
    classDef core fill:#1a1a30,stroke:#ffb450,color:#e8eef5
    classDef dep fill:#0e2030,stroke:#5eead4,color:#e8eef5
    class IN in
    class LS,CB gate
    class ORD,BH core
    class DEP,FB dep

Each dependency is isolated; failures fail fast to a fallback so one slow dep can't exhaust global resources

Key Techniques

1. Circuit Breaker — pull the switch on a failing dependency

One-line trade-off: trade "temporary unavailability of the failing dependency" for "the caller fails fast and the downstream gets to breathe".

Principle: the electrical-breaker metaphor. A three-state machine: Closed (let traffic through, track failure rate) → failure rate exceeds threshold → Open (reject directly, send nothing, go straight to fallback) → cool down → Half-Open (admit a few probe requests) → success returns to Closed, failure goes back to Open. It turns "wait out the full timeout before failing" (wasting seconds each time while still piling pressure on a dying downstream) into fail fast, and by cutting the sustained barrage it gives the downstream a chance to recover.

# Circuit breaker state machine (pseudo-code)
def call(dep, req):
    if state == OPEN:
        if now() - opened_at > cooldown:   # cooldown elapsed
            state = HALF_OPEN               # admit one probe
        else:
            return fallback(req)            # fail fast, don't hit dep
    try:
        resp = dep.invoke(req, timeout=200ms)
        on_success()                        # half-open success -> CLOSED
        return resp
    except (Timeout, ServerError):
        on_failure()                        # window error rate over threshold -> OPEN
        return fallback(req)

2. Retry with Backoff + Jitter — a double-edged sword

One-line trade-off: trade "extra load from retries" for "absorbing transient blips" — but get it wrong and you pour gasoline on the fire.

Principle: transient faults (network blips, leader election, occasional timeouts) usually heal with one retry. But naive immediate retry is an amplifier: downstream overloads → many requests fail → all clients retry at once → downstream gets hit with double the traffic and dies for good. That's a retry storm. Two antidotes: ① exponential backoff, doubling the wait after each failure (100ms→200ms→400ms) to give the downstream room to recover; ② jitter, adding randomness so retry times spread out — otherwise clients "align" into a synchronized pulse and slam the downstream periodically.

# Full-jitter exponential backoff (AWS recommended)
def retry(req, max_attempts=3):
    base, cap = 0.1, 2.0          # 100ms base, 2s cap
    for attempt in range(max_attempts):
        try:
            return call(req)       # only retry idempotent + retryable errors
        except Retryable:
            if attempt == max_attempts - 1: raise
            if not retry_budget.try_acquire(): raise  # global retry budget
            backoff = min(cap, base * 2 ** attempt)
            sleep(random.uniform(0, backoff))         # full jitter

3. Bulkhead — partition resources into watertight compartments

One-line trade-off: trade "lower overall resource utilization" for "a single dependency's failure can't exhaust global resources".

Principle: from shipbuilding — a hull split into watertight compartments, so a breach in one doesn't sink the ship. Mapped to systems: don't let all downstream calls share one thread/connection pool. Give each dependency its own quota (e.g. coupons gets at most 20 threads). Then if coupons slows down, it can only fill its own 20, while critical inventory and payment calls still have threads — the failure is sealed in its compartment. This is the real cure for the avalanche scenario at the top.

# Semaphore bulkhead: cap per-dependency concurrency (pseudo-code)
sem = {dep: Semaphore(limit[dep]) for dep in deps}   # per-dep quota
def call(dep, req):
    if not sem[dep].try_acquire():      # full -> reject immediately, no queue
        return fallback(req)            # compartment full -> doesn't spill to others
    try:
        return dep.invoke(req, timeout=200ms)
    finally:
        sem[dep].release()

4. Graceful Degradation + Load Shedding — partial beats total death

One-line trade-off: trade "feature completeness / result precision" for "keeping the critical path alive".

Principle: when a dependency fails or the system overloads, rather than returning 500 or collapsing entirely, return a degraded response: coupon service down → place the order at list price (fall back to a default); personalized recommendations down → return the popular list (fall back to cache/static). The companion at ingress is load shedding: as the system nears overload, proactively reject some requests (return 429) so the rest can complete normally — partial success beats everyone timing out. Choose what to drop by criticality: shed non-critical first, protect core checkout.

Extensions & Optimizations

• Adaptive concurrency limits: instead of hand-tuning fixed bulkhead sizes, use a TCP-congestion-control-like algorithm (Netflix concurrency-limits) that adjusts the concurrency cap based on latency changes.
• Hedged requests: for idempotent reads, if p95 hasn't returned, fire a second request in parallel and take the first to arrive — sharply cutting tail latency (pair with a retry budget to prevent amplification).
• End-to-end timeout budget: allocate a total budget at ingress (e.g. 300ms) and propagate it decreasing per hop (deadline propagation), so each downstream knows how much time is left and gives up when it's gone.
• Chaos engineering: writing these protections doesn't mean they work — use fault injection (inject latency, kill deps) to verify the breaker actually trips and the fallback actually catches (see the upcoming chaos-engineering issue).
• Regional failover: when an AZ/region fails, shift traffic to a healthy region — reliability one level above per-host isolation.

Common Pitfalls + Interview Follow-ups

Further Resources

• Amazon Builders' Library: Marc Brooker, Timeouts, retries, and backoff with jitter — the authoritative practice on timeouts/retries/backoff.
• Google SRE Book: Handling Overload and Addressing Cascading Failures — load shedding, throttling, cascading failures.
• Martin Fowler bliki: CircuitBreaker — the standard definition.
• Netflix: Hystrix Wiki — engineering details of circuit breaker + bulkhead + fallback.
• Michael Nygard, Release It! (Pragmatic Bookshelf): the original source of the stability patterns (Circuit Breaker / Bulkhead / Fail Fast / Shed Load).

Deeper Thinking (click to expand)