Distributed Transactions — Don't. And if you must, keep them tiny.2PC / 3PC · Saga (Orchestration vs Choreography) · Transactional Outbox · Idempotency · Why Microservices Avoid Distributed Transactions

Scenario & Constraints

Design a cross-border e-commerce checkout flow. A single checkout must complete 5 things at once:

1. Order service writes the order (own Postgres)
2. Inventory service decrements stock (own Postgres)
3. Payment service calls Stripe to charge (external API, not rollbackable)
4. Points service deducts loyalty points (own DynamoDB)
5. Notification service sends email + push (external SES / FCM)

Scale: 5k peak TPS checkout, P99 ≤ 800ms end-to-end, 0 oversells, 0 duplicate charges. The 5 services are owned by 5 teams with no shared database. Every intermediate step may fail: service crash, network partition, Stripe 504, message loss.

Question: how do these 5 steps guarantee "all succeed or none happen (or compensate observably)"? This is the distributed transactions problem. Today: why textbook 2PC/3PC is almost never used in production, how Saga actually lands, how Outbox solves dual-write, and why "the best distributed transaction is none at all."

High-level Architecture

graph TD
    Client["Client"]
    Orch["Saga Orchestrator
Temporal / Cadence
persists every step"]

    subgraph S1["Order Domain"]
        OrdSvc["Order Service"]
        OrdDB[("Postgres
orders + outbox")]
    end
    subgraph S2["Inventory Domain"]
        InvSvc["Inventory Service"]
        InvDB[("Postgres
reservations + outbox")]
    end
    subgraph S3["Payment Domain"]
        PaySvc["Payment Service"]
        Stripe["Stripe API
external, not rollbackable"]
    end
    subgraph S4["Points Domain"]
        PtsSvc["Points Service"]
        PtsDB[("DynamoDB
+ idempotency table")]
    end
    subgraph S5["Notification Domain"]
        Notify["Notify Service"]
    end

    CDC["Debezium CDC
reads outbox table"]
    Kafka["Kafka
order.events"]

    Client --> Orch
    Orch -->|1 createOrder| OrdSvc
    Orch -->|2 reserveStock| InvSvc
    Orch -->|3 charge| PaySvc
    Orch -->|4 deductPoints| PtsSvc
    Orch -.->|5 async notify| Kafka
    OrdSvc --> OrdDB
    InvSvc --> InvDB
    PaySvc --> Stripe
    PtsSvc --> PtsDB

    OrdDB -.-> CDC
    InvDB -.-> CDC
    CDC --> Kafka
    Kafka --> Notify

    classDef orch fill:#2a1530,stroke:#ff7ab6,color:#e8eef5
    classDef ext fill:#3a2010,stroke:#ffb450,color:#e8eef5
    class Orch orch
    class Stripe ext

Core idea: a Saga (Temporal/Cadence) chains the 5 steps; each is a local transaction with a reverse compensation on failure. Event fan-out goes through Outbox + CDC (DB commit ⇒ Kafka emit is guaranteed). Weak-transaction work like notifications goes eventual. The whole system has zero 2PC.

Key Techniques

1. 2PC / 3PC: textbook answer, production liability

Mechanism: Two-Phase Commit uses a coordinator to orchestrate multiple participants:

• Phase 1 (prepare): coordinator asks all participants "can you commit?"; each writes a prepare log + locks resources, replies yes/no.
• Phase 2 (commit/abort): all yes → commit all; any no → abort all.

Correctness gives ACID atomicity across nodes. But:

# 2PC disaster — why Pat Helland called it a "Maginot Line"
# T = 0    coord -> all: PREPARE
# T = 10ms all -> coord: YES (each locks resources, writes prepare log)
# T = 11ms coord crashes ⚠
# T = ∞    participants hold locks waiting for coord to recover
#          other transactions blocked, business halts
#
# Even when coord restarts, it must replay its transaction log
# before deciding; participants stay locked. This is the worst
# failure mode in distributed systems: "one process death stalls
# the entire system."

2. Saga: break one big transaction into N local transactions + N-1 compensations

Origin: Garcia-Molina & Salem introduced Saga at SIGMOD 1987, originally for "long-running transactions inside a single DB" (to avoid long-held locks). Microservices adopted it: split a cross-service transaction into a chain of local transactions T1…Tn; if any Ti fails, execute compensations C(i-1)…C1 in reverse order for completed steps.

You lose ACID isolation but gain eventual consistency + business-level reversibility. Two landing patterns:

sequenceDiagram
    participant C as Client
    participant O as Orchestrator (Temporal)
    participant Ord as Order Svc
    participant Inv as Inventory Svc
    participant Pay as Payment (Stripe)
    participant Pts as Points Svc

    C->>O: placeOrder()
    O->>Ord: 1 createOrder (PENDING)
    Ord-->>O: orderId=42
    O->>Inv: 2 reserveStock(sku, qty)
    Inv-->>O: reserved
    O->>Pay: 3 charge($100, idempKey=42)
    Pay-->>O: paid (charge_xyz)
    O->>Pts: 4 deductPoints(user, 50, idempKey=42)
    Pts--xO: fail! insufficient points
    Note over O: Saga reverse compensations
    O->>Pay: C3 refund(charge_xyz, idempKey=42-refund)
    O->>Inv: C2 releaseStock(sku, qty, idempKey=42-rel)
    O->>Ord: C1 markFailed(42, reason="points")
    O-->>C: failed: insufficient points, fully rolled back

# Temporal Workflow-style pseudocode (Python SDK)
@workflow.defn
class PlaceOrderSaga:
    @workflow.run
    async def run(self, req: OrderReq) -> Result:
        compensations = []          # stack of comp actions for completed steps
        try:
            order = await activity.execute(create_order, req)
            compensations.append(lambda: mark_failed(order.id))

            await activity.execute(reserve_stock, req.sku, req.qty)
            compensations.append(lambda: release_stock(req.sku, req.qty))

            charge = await activity.execute(
                stripe_charge,
                amount=req.amount,
                idempotency_key=f"order-{order.id}"   # key: idempotency
            )
            compensations.append(
                lambda: stripe_refund(charge.id, key=f"refund-{order.id}")
            )

            await activity.execute(deduct_points, req.user, req.points,
                                   key=f"pts-{order.id}")
            return Result.ok(order.id)

        except ActivityError as e:
            # Run compensations in reverse. Temporal guarantees at-least-once + retry.
            for comp in reversed(compensations):
                await activity.execute(comp, retry_policy=infinite_backoff)
            return Result.fail(reason=str(e))

# Key properties:
#  - Temporal persists workflow state to an event log; on crash it replays and resumes.
#  - Every activity must be idempotent (because it may retry).
#  - Compensations must be idempotent AND not allowed to fail forever
#    (a failed compensation needs human intervention).

3. Outbox Pattern: kill "dual write," demote distributed transactions to local ones

Problem: a service must "write DB" + "publish Kafka event." The two are not in one transaction, so there is always an inconsistency window:

• DB succeeds, Kafka fails → downstream never receives → data loss
• Kafka succeeds, DB fails (power loss) → downstream receives an event for an order that doesn't exist → ghost orders

This is the dual write problem, the most common transactional pitfall in microservices.

Outbox solution:

1. The business table and the outbox table are written in the same local DB transaction. Local = ACID; both succeed or both fail.
2. A separate relay process (Debezium-style CDC reading the WAL/binlog) asynchronously ships new outbox rows to Kafka.
3. The relay tracks its own offset, delivers at-least-once; consumers dedupe on outbox row ID.

graph LR
    App["Order Service"]
    DB[("Postgres
orders + outbox
same local txn")]
    Debez["Debezium
reads WAL"]
    K["Kafka
order.events"]
    Cons1["Inventory Consumer"]
    Cons2["Analytics Consumer"]
    Cons3["Notification Consumer"]

    App -- "BEGIN
INSERT orders
INSERT outbox
COMMIT" --> DB
    DB -. WAL .-> Debez
    Debez --> K
    K --> Cons1
    K --> Cons2
    K --> Cons3

# Outbox in practice — Postgres + Debezium
CREATE TABLE orders (id BIGSERIAL PK, user_id ..., amount ..., status ...);
CREATE TABLE outbox (
    id BIGSERIAL PK,
    aggregate_id BIGINT,        -- order id (used as partition key)
    event_type TEXT,            -- "order.created" / "order.cancelled"
    payload JSONB,
    created_at TIMESTAMPTZ DEFAULT now()
);

-- Business code: write orders and outbox event in the same txn
BEGIN;
  INSERT INTO orders (...) VALUES (...) RETURNING id;
  INSERT INTO outbox (aggregate_id, event_type, payload)
       VALUES (42, 'order.created', '{"user":7,"amount":100}');
COMMIT;        -- atomic: both succeed or neither does

-- Debezium attaches to the Postgres replication slot,
-- tails the WAL → watches outbox INSERTs → transforms → pushes Kafka.
-- The business code doesn't need to know Kafka exists — fully decoupled.

# Why not CDC on the business table directly?
#  - Business schema changes often; downstreams become tightly coupled.
#  - A single row change may correspond to 0 or many domain events
#    (an order INSERT may emit "order.created" + "loyalty.points.eligible").
#    Outbox lets business code control event granularity.

# Why not just call Kafka producer in the same function?
#  - Producer.send() is not in the DB transaction — dual write problem.
#  - Even with Kafka transactions, you cannot span Postgres + Kafka.

4. Idempotency: the last line of defense for distributed transactions

Mechanism: in distributed systems every RPC, message, and callback is at-least-once — a network timeout can't distinguish "request lost" from "response lost," so retries are inevitable. Idempotency guarantees "executing the same operation N times has the same effect as executing it once" — it is the safety net for saga retries, outbox redelivery, and compensations.

# Stripe-style idempotency key, server-side (Postgres)
# Reference: https://stripe.com/blog/idempotency

CREATE TABLE idempotency (
    key TEXT PRIMARY KEY,
    request_hash TEXT NOT NULL,     -- detect key reuse with different bodies
    response_code INT,
    response_body JSONB,
    status TEXT,                    -- 'in_progress' | 'completed'
    locked_at TIMESTAMPTZ,
    created_at TIMESTAMPTZ DEFAULT now()
);

def handle(key, req):
    h = hash(req.body)
    row = db.fetch_for_update("SELECT * FROM idempotency WHERE key=%s", key)

    if row and row.request_hash != h:
        raise Conflict("idempotency key reused with different body")

    if row and row.status == 'completed':
        return row.response_body                # return cached response

    if row and row.status == 'in_progress':
        if now() - row.locked_at > LEASE:
            pass                                # lease expired, retry ok
        else:
            raise Conflict("request in flight")

    # claim
    db.upsert("idempotency", key=key, request_hash=h,
              status='in_progress', locked_at=now())

    try:
        resp = do_business(req)
        db.update("idempotency", key=key,
                  status='completed', response_code=200,
                  response_body=resp)
        return resp
    except RetryableError:
        # don't update; let client retry
        raise
    except FatalError as e:
        db.update("idempotency", key=key,
                  status='completed', response_code=500,
                  response_body={"error": str(e)})
        raise

# Expiry policy: Stripe keeps keys for 24h, then prunes. Clients should
# not assume keys live forever.

Extension & Optimization

• Saga + workflow engine is the de facto standard: for cross-service flows of more than 3 steps, do not hand-roll state machines. Adopt Temporal / Cadence / AWS Step Functions / Azure Durable Functions — you get persistence, retry, visualization, and versioning for free.
• Outbox alternatives: CockroachDB / TiDB changefeeds stream directly to Kafka, skipping Debezium; MongoDB / DynamoDB change streams do the same. For new projects, prefer databases with built-in CDC.
• Semantic Locking: during a saga, data may be visible to other transactions (isolation violated). Chris Richardson proposes semantic locks — mark data with a business-level 'pending' flag; other transactions see pending and decide what to do (reject / wait / force).
• Inbox pattern (outbox dual): the consumer keeps an inbox table of processed event IDs; "write business table + write inbox" is one local transaction. Consumer dedup becomes a local-transaction problem.
• Transaction ID / trace propagation: propagate saga_id / step_id across services; combined with OpenTelemetry it gives full traceability — when a saga is stuck you locate the step in seconds.
• The fundamental weapon against distributed transactions: domain boundaries. If 90% of flows cross the same two services, they probably should be one service. Wrong microservice boundaries are what force you into distributed transactions in the first place.

Common Pitfalls & Interview Questions

Interview follow-ups:

1. Walk through 2PC blocking: coordinator dies after phase-1 ACK — what do participants do? How does 3PC improve and why is it still rare?
2. How does Saga differ from ACID in isolation? Give a bug caused by Saga's visible intermediate state.
3. Outbox vs Kafka transactions — why is Outbox preferred in production?
4. Design "cross-service transfer": service A debits 100, service B credits 100. Give the saga + compensation flow and failure-mode analysis.
5. What's the expiry policy for idempotency keys? What happens if a client reuses the same key after expiry? How do you avoid double charges?
6. "Should microservices avoid distributed transactions?" — how do you answer? Give an example of "converting a distributed transaction into a domain redesign."

Deep Resources

• "Designing Data-Intensive Applications," Ch 7 §Transactions + Ch 9 §Atomic Commit and Two-Phase Commit (Kleppmann): the most complete industry-grade treatment of 2PC and the limits of distributed transactions.
• Pat Helland — "Life Beyond Distributed Transactions: An Apostate's Opinion" (ACM Queue 2017 / CIDR 2007): the consensus-shaping paper. Established the saga / messaging / entity school.
• Garcia-Molina & Salem — "Sagas" (SIGMOD 1987): the original saga paper, 14 pages. Reading it gives you a deeper grasp than any modern framework's docs.
• Chris Richardson — microservices.io: Saga Pattern + "Microservices Patterns" (Manning): includes orchestration / choreography comparison, semantic lock, saga versioning, and operational lessons.
• Debezium — "Reliable Microservices Data Exchange With the Outbox Pattern" (2019): the canonical outbox + CDC reference.
• Stripe — "Designing robust and predictable APIs with idempotency" (2017) + Brandur Leach's idempotency keys in Postgres: best engineering references.
• Uber Cadence — "Announcing Cadence 1.0" (2024); DoorDash — "Building Reliable Workflows with Cadence" (2024): saga orchestration at hyperscale.
• Temporal — "The Definitive Guide to Durable Execution": theory and practice for modern saga + durable workflow platforms.

Deep Reflection