File & Object Storage — Trading 1.5x Space for Eleven 9s of DurabilityS3 Architecture, Erasure Coding, Multipart Upload, Durability

Problem & Constraints

Design an S3-like object store: serving hundreds of billions of objects, 100 PB, objects from 1 byte to 5 TB, with eleven 9s of durability (99.999999999% — fewer than 1 lost object per 10 million per year). This is the bedrock under cloud drives, backups, data lakes, and image/video origins — Dropbox, Netflix masters, and your app's avatars all live here.

• Read/write ratio & size distribution: write-once-read-many, but object size is extremely long-tailed — 99% of objects are small (<1MB), 99% of bytes live in big files. Optimize both ends separately.
• Consistency: a GET right after a PUT must see it (read-after-write); overwrite and list consistency can be relaxed.
• Durability ≫ availability: lost data is irreversible; a brief outage is recoverable. Durability is the first-class citizen at eleven 9s; four 9s of availability is fine.
• Throughput: a single 5 TB object takes an hour even at 10 Gbps — you need parallel chunking + resumable upload, not one TCP stream end to end.
• Cost: 100 PB at 3x replication is 300 PB of raw disk; erasure coding squeezes it to ~130 PB. Cost per PB per month (power, racks) is life or death.

High-Level Architecture

graph TD
    C["Client
SDK / direct PUT via presigned URL"]
    LB["API layer
S3 protocol / Auth / throttle"]
    subgraph CP["Control Plane"]
      META["Metadata service
key → chunk locations, sharded KV"]
    end
    subgraph DP["Data Plane"]
      PL["Placement / EC encoder"]
      S1["Storage Node 1"]
      S2["Storage Node 2"]
      S3["Storage Node N
raw disk + CRC"]
    end
    BG["Background jobs
scrub · repair · GC · tiering"]
    C --> LB
    LB --> META
    LB --> PL
    PL --> S1 & S2 & S3
    META -.index.-> DP
    BG -.scan/repair.-> DP

The core is control-plane / data-plane separation: the metadata service holds the "object key → which disks hold the chunks" index (small, strongly consistent, must scale to hundreds of billions of rows); the data plane just stores immutable byte chunks (large, optimized for throughput and cost). Scaling them independently is what sets object storage apart from a file system — no directory tree, no inode chains; a flat keyspace makes horizontal scaling almost unbounded. Background jobs (scrub, repair, GC) are the true source of durability, decoupled from the request path.

Key Techniques

1. Three Storage Models — Why Object Storage Scales Without Limit

The trade-off in one line: object storage gives up in-place mutation and directory semantics in exchange for a flat keyspace's unbounded horizontal scale and rock-bottom unit cost.

Principle: Block (e.g. EBS/raw disk) gives fixed-size sectors — most flexible but you manage the filesystem yourself; File (e.g. NFS/EFS) gives a POSIX directory tree and in-place rewrites, but the directory tree is a tree that must be kept consistent, so metadata becomes the scaling bottleneck. Object (e.g. S3) treats the whole file as an immutable blob keyed by a flat string (the `/` in `a/b/c.jpg` is just a character, not a directory) — no rename, no append, no inode chain. Metadata collapses into a giant KV table that can be sharded arbitrarily by key hash.

2. The Durability Engine — Replication vs Erasure Coding

The trade-off in one line: 3 replicas are simple and repair fast but cost 200% space amplification; erasure coding drops to ~30-50% amplification at the cost of encoding CPU, read amplification during repair, and being unfit for small files.

Principle: for eleven 9s, a single disk's ~2-4% annual failure rate is nowhere near enough. 3-replication puts one copy in each of 3 AZs — any 2 can fail and you still read; brute-force simple. Erasure coding (Reed-Solomon) splits the object into k data shards, computes m parity shards, and spreads all k+m across distinct fault domains; any k shards reconstruct the object. Backblaze Vault uses 17+3: 17 data + 3 parity = 20 shards, lose any 3 and lose nothing, with only 20/17≈1.18x amplification — and it too reaches eleven 9s.

# Reed-Solomon idea (pseudo-code, not runnable)
# treat data shards as polynomial coefficients;
# parity = evaluations at extra points
data   = split(obj, k=17)              # 17 data shards
parity = rs_encode(data, m=3)          # Vandermonde/Cauchy matrix → 3 parity
shards = data + parity                 # 20 shards across 20 fault domains

# rebuild after losing <=3 shards: solve a 17-unknown linear system
def repair(surviving_shards):          # any 17 suffice
    return rs_decode(surviving_shards) # matrix inverse x survivors

3. Large-Object Upload — Multipart + Direct Client Upload

The trade-off in one line: parallel chunked upload + presigned-URL direct upload buys high throughput and resumability, at the cost of client complexity and "orphan parts" needing GC.

Principle: 5 TB over a single TCP stream restarts from zero on any hiccup, and single-stream throughput is capped by RTT × window. Multipart upload splits the object into parts (S3 mandates 5MB–5GB/part, max 10,000 parts); each part uploads independently — parallel, retryable, with its own ETag — and a final CompleteMultipartUpload stitches the part list into the object. Paired with a presigned URL: the server signs a time-limited URL with its secret, and the client PUTs directly to the storage layer, bytes never touching your app servers — saving bandwidth and a network hop.

# the server only issues signatures, never touches the byte stream
def init_upload(key, size):
    upload_id = meta.create_multipart(key)
    n = ceil(size / PART_SIZE)                 # e.g. 64MB/part
    urls = [presign("PUT", key, upload_id, i)  # one signed URL per part
            for i in range(n)]
    return upload_id, urls
# client: PUT parts in parallel, retry only the failed part, record {part_i: etag}
# finish: POST complete(upload_id, [(i, etag)...]) -> atomic metadata commit

4. Strong Metadata Consistency & Hot Partitions

The trade-off in one line: strong consistency makes read-after-write easy, but metadata must be sharded to handle hundreds of billions of keys, and sharding creates hot spots on sequential keys.

Principle: metadata is a giant table of "key → shard locations + version + ACL" and must be sharded (by key hash). Before 2020 S3 was eventually consistent (a freshly PUT object might not appear in a list); in late 2020 S3 shipped strong read-after-write consistency — once a PUT succeeds, any GET/LIST sees it immediately, at no extra price or performance cost. The price is that the metadata layer must be strongly consistent within a shard (typically a Paxos/Raft-replicated KV). But sharding by sequential key prefix hits hot spots: if every key is a `2026-06-20/...` time prefix, writes forever slam the same shard.

Scaling & Optimization

• Tiering: hot data on SSD/replicas, warm on HDD/EC, cold on SMR/tape (S3 Glacier). Auto-demote by access frequency to cut cost by an order of magnitude. Dropbox specifically optimized Magic Pocket for cold storage and adopted SMR drives.
• Versioning + soft delete: immutable objects are naturally multi-versioned; delete just writes a tombstone, with GC reclaiming later. This is also the basis for backup and ransomware defense (versioning + MFA delete + cross-region replication).
• Continuous scrubbing: background jobs constantly recompute and compare CRCs to catch bit rot and bad disks early, repairing before data becomes unrecoverable. This — more than the request path — is what actually buys eleven 9s.
• Cross-region replication (CRR): asynchronously copy objects to another region for DR and data-residency compliance. Note it is eventually consistent; RPO is not zero.

Pitfalls + Interview Questions

• Treating object storage as a filesystem: assuming `ls` on a "directory" is cheap — it's actually a LIST prefix scan, slow and pricey at millions of keys. There's no true rename (it's copy+delete).
• Replicating cold data: 3 replicas for cold data burns money — use EC; conversely, EC for small files is asking for trouble.
• Forgetting orphan multiparts: skip the lifecycle abort rule and cost leaks month after month.
• Follow-ups: ① How is eleven 9s actually computed? (a probability model over fault-domain independence + repair speed vs failure rate) ② How do you upload a 5 TB object? Resume it? ③ What does each of strong vs eventual consistency give up in object storage? ④ How do you handle hot keys — is it the same problem as database sharding? ⑤ When is space actually freed after deleting an object? (GC + tombstone)

Deep-Dive Resources

• "Designing Data-Intensive Applications" (Kleppmann) — the underlying principles of replication, partitioning, and consistency; the metadata layer is exactly these concepts applied.
• Werner Vogels — "Building and operating a pretty big storage system called S3" (allthingsdistributed.com, 2023) — a first-hand view of S3's durability, scrubbing, and scale.
• Dropbox Engineering — "Inside the Magic Pocket" (dropbox.tech) — EC, placement, and migration in an exabyte-scale self-built object store.
• Backblaze — "Reed-Solomon Erasure Coding" / "Vault Architecture" (backblaze.com/blog) — 17+3 EC and an open-source implementation, the durability math made concrete.

Going Deeper