Agent Error Recovery & Resilience

The Two Layers of Retry: don't mix transport errors and semantic errors

Background & Principle

There are two kinds of "failure" in an agent, handled in opposite ways.

• Transport failure: HTTP 429 / 503, connection timeout, ECONNRESET — deterministic transient faults. The right move is a program-level retry with exponential backoff + jitter; the model should never even hear about it. Marc Brooker's classic AWS post shows that fixed backoff makes clients "synchronize" and hammer the same instant; full jitter (randomizing the whole backoff window) sharply cuts contention.
• Semantic failure: the tool ran but the result is wrong — file not found, command errored, invalid argument. This is information the model needs to see and self-correct; feed the error back as a tool_result and let it try another way. Never retry it in program code — the same arguments fail the same way a hundred times.

The #1 bug is swapping the layers: treating a 429 as semantic and feeding it to the model (which won't back off and keeps hammering the dependency), or treating "file not found" as transport and retrying in code (infinite loop). The test is one sentence: is the error input-independent and merely transient? Yes → program backoff; No → feed back to the model.

Hands-on

import time, random, anthropic

TRANSIENT = (anthropic.RateLimitError, anthropic.APITimeoutError,
             anthropic.InternalServerError)

def call_with_backoff(fn, max_retries=5):
    for attempt in range(max_retries):
        try:
            return fn()
        except TRANSIENT as e:        # transport: deterministic transient → program backoff
            if attempt == max_retries - 1: raise
            # full jitter: randomize the whole window, avoid thundering herd
            time.sleep(random.uniform(0, min(60, 2 ** attempt)))
        # note: semantic errors (tool failures) are NOT caught here —
        # they go back as tool_result; the model decides what's next

Idempotency & Compensation: agents will repeat, so side-effecting tools must be idempotent

Background & Principle

Retry and resume share a hidden premise: the operation being retried can be run multiple times safely. Reads are naturally idempotent (reading a file twice is fine), but writes are not: create_order / send_email / charge_card run once more = the customer charged twice. Agents are more dangerous than traditional systems because "retry" has two sources: program-level backoff, and the model — when a tool_result is ambiguous (a timeout, did it succeed?) the model "calls it again just to be safe." That's exactly the riskiest moment, because the first call may already have succeeded.

Two engineering tools:

• Idempotency key: every side-effecting tool call carries a client-generated unique key; the server dedupes on it — a second call with the same key returns the first result without re-executing. Standard practice in payment APIs like Stripe. The key must be derived from the task (deterministic), not random per call, or a retry's changed key defeats dedup.
• Compensation (Saga): agents have no DB transaction to roll back with one call. If step 3 fails, the side effects of steps 1–2 (booked, charged) must be undone by explicit compensating actions (cancel, refund). Pair every side-effecting step with a compensating action, run them in reverse on failure.

Hands-on

# 1) idempotency key: makes retry safe. derive from task, not random
def book_flight(args):
    key = f"book:{args['trip_id']}:{args['flight_no']}"   # deterministic key
    return api.post("/bookings", json=args,
                    headers={"Idempotency-Key": key})       # server dedupes

# 2) compensation: register an undo per side-effect; reverse on failure
saga = []
try:
    r1 = book_flight(...);  saga.append(lambda: cancel_flight(r1.id))
    r2 = charge_card(...);  saga.append(lambda: refund(r2.id))
    r3 = book_hotel(...)                  # ← this step fails
except Exception:
    for undo in reversed(saga):     # reverse: refund first, then cancel
        undo()
    raise

Checkpoint & Resume: agent state must live outside the process — an in-memory loop is a toy

Background & Principle

An agent runs 40 steps over half an hour and crashes at step 38 (OOM, deploy restart, a spot machine reclaimed). If state lives only in the in-memory messages array, the prior 37 steps are wasted — and their tokens and side effects are already spent. A production agent must persist state outside the process: checkpoint after each step, and on crash resume from the last checkpoint. This is the core of durable execution. Two ways to land it:

• Library level: LangGraph's checkpointer snapshots state to SQLite/Postgres after each node and resumes by thread_id — "loads the last snapshot and resumes exactly where it left off."
• Platform level: a durable execution engine like Temporal turns the whole agent workflow into something crash-recoverable — on crash it replays history and skips completed activities, "survive a crash and resume exactly where it left off."

The key is checkpoint granularity: too fine (every token) is costly, too coarse (only at the end) is useless. In practice, checkpoint per model call or per side-effecting step. And it must pair with idempotency (§2) — on resume, that last step may have run only halfway.

Hands-on

import json, os
def agent(task, ckpt_path):
    if os.path.exists(ckpt_path):                    # resume: restore from checkpoint
        s = json.load(open(ckpt_path)); msgs, step = s["msgs"], s["step"]
    else:
        msgs, step = [{"role":"user","content":task}], 0
    while step < MAX_STEPS:
        r = client.messages.create(model=MODEL, messages=msgs,
                                   tools=SCHEMAS, max_tokens=4096)
        msgs.append({"role":"assistant","content":r.content})
        if r.stop_reason == "end_turn": break
        results = run_tools(r.content)               # run tools (must be idempotent!)
        msgs.append({"role":"user","content":results}); step += 1
        json.dump({"msgs":msgs,"step":step}, open(ckpt_path,"w"))  # persist per step

Timeout, Circuit Breaker & Partial Failure: distinguish "degradable" from "must stop"

Background & Principle

Three related mechanisms:

• Timeout: every tool call needs an upper bound, or one hung HTTP request stalls the whole agent forever. Timeouts should be layered: per-tool < per-step < a top-level task deadline.
• Circuit breaker: when a downstream (an API, an MCP server) fails N times in a row, it "trips" — subsequent calls fail fast instead of waiting on timeouts, then "half-open" lets one probe through after a cooldown. This stops the agent from repeatedly hammering a dead dependency, saving tokens and avoiding cascading slowdowns. The classic three states: closed → open → half-open.
• Partial failure: in a parallel fan-out (Day 13's orchestrator-worker), 5 workers run and 1 dies. The anti-pattern is failing the whole batch; the mature move is to collect the 4 successes and hand "1 failed" to the orchestrator as information — degrade and continue with 4, or treat the subtask as critical and stop.

Hands-on

class CircuitBreaker:
    def __init__(self, fail_max=5, reset_after=30):
        self.fail_max, self.reset_after = fail_max, reset_after
        self.fails, self.opened_at = 0, None
    def call(self, fn):
        if self.opened_at:                          # OPEN
            if time.time() - self.opened_at < self.reset_after:
                raise CircuitOpen("dependency tripped, fail fast")
            self.opened_at = None                   # → HALF-OPEN: let one probe through
        try:
            out = fn(); self.fails = 0              # success → CLOSED
            return out
        except Exception:
            self.fails += 1
            if self.fails >= self.fail_max:
                self.opened_at = time.time()       # → OPEN: trip
            raise