AI Product Backend — RAG, Agentic Loops & Human-in-the-loopRAG, Durable Agent Loops, Embedding Serving, Human-in-the-loop

Scenario + Requirements

Build a "document Q&A + agent assistant" backend for 500K enterprise users: users ask questions, the system retrieves from an internal knowledge base to answer; complex tasks ("compare these three contracts into a table and email it") go to an agent for multi-step execution. This is not "wrap an LLM API"—it is simultaneously a retrieval system, a long-running task orchestrator, and an exercise in fencing a non-deterministic model output inside controllable engineering boundaries.

• Scale: 100M chunks (~20M source docs), 1024-dim vectors; Q&A peak 2000 QPS, agent task concurrency 5000.
• Latency: Q&A p95 first token < 1.5s (retrieval budget ≤ 300ms); agent tasks may run seconds to minutes but must be resumable.
• Correctness: answers must be attributable (traceable to a chunk); high-risk actions (send email, mutate data) require human confirmation.
• Cost: LLM tokens dominate—the longer the retrieved context, the more expensive. Recall quality converts directly into dollars.

High-level Architecture

Three main flows: ① the Q&A hot path runs "retrieve → fuse → rerank → stream", on a tight budget; ② the agent path runs on a durable execution engine that can run long and resume after interruption; ③ offline ingest turns documents into searchable vectors+inverted index, decoupled from online. Four key points below.

Key Technical Points

① RAG retrieval quality — bad recall can't be saved by a smarter model

One-line trade-off: naive chunk-embedding is cheap but recalls poorly; contextual + hybrid + rerank recalls well but is costly and slower.

[Principle] The RAG bottleneck is almost never generation—it's retrieval recall. The model can only answer from the chunks you feed it; what isn't retrieved doesn't exist. The naive approach splits docs into 512-token chunks and embeds each in isolation, but chunking strips context: a passage says "this approach cut latency 40%", yet which approach, in which doc, is invisible from the isolated chunk—so semantic search misses. Anthropic's Contextual Retrieval prepends an LLM-generated "what this chunk is about" summary before embedding, and adds BM25 keyword retrieval for hybrid recall.

# Online retrieval: hybrid recall + rerank (pseudocode)
q_vec = embed(query)                      # vectorize query
dense = vdb.ann_search(q_vec, k=100)      # semantic recall top-100
sparse = bm25.search(query, k=100)        # keyword recall top-100
fused = rrf(dense, sparse)                # reciprocal rank fusion, dedup
top = cross_encoder.rerank(query, fused)[:8]  # keep 8 most relevant
context = "\n\n".join(c.text for c in top)
answer = llm.stream(prompt(query, context))   # stream w/ citations

② Agent loop as a service — turn a long, crash-prone loop into a resumable workflow

One-line trade-off: an in-memory loop is simple but loses everything on crash; durable execution is resumable but must persist every step.

[Principle] An agent is a loop: call model → model requests a tool → run the tool → feed result back → call model again, until done. The problem: this loop may run minutes and dozens of steps, during which any step can hit a deploy restart, tool timeout, or model 503. If state lives only in memory, one crash restarts the whole task from scratch—after burning real tokens on every step. The fix is durable execution: persist each step's input/output as an event, and on crash replay from the last successful step without re-running completed tool calls (via idempotency keys). This is exactly the value of engines like Temporal / AWS Step Functions.

# Durable agent loop (durable workflow pseudocode)
def agent_workflow(task):
    history = [user_msg(task)]
    for step in range(MAX_STEPS):             # hard cap against runaway
        resp = activity.call_llm(history)     # persisted; replay won't recompute
        if resp.is_final:
            return resp.text
        for call in resp.tool_calls:
            # idempotency key = (workflow_id, step, tool, args_hash)
            result = activity.run_tool(call, idem_key=key(step, call))
            history.append(tool_result(call, result))
    raise StepLimitExceeded   # trigger degrade / handoff to human

③ Embedding service design — model version and index version must be locked together

One-line trade-off: online real-time embedding is low-latency but pricey, batch embedding is high-throughput but delayed; the biggest trap is changing the model = re-embedding the whole corpus.

[Principle] The embedding service has two traffic classes: online (query vectorization, must be fast, cacheable) and offline (batch-embedding 100M chunks, needs throughput). They should be deployed separately: online for small low-latency batches, offline for large GPU-saturating batches. The key constraint is the same vector space—query and doc must be embedded by the same model, or vectors can't be compared. So once you upgrade the embedding model, the entire index must be rebuilt with the new model, with old and new coexisting during a gradual cutover—never mixed mid-flight (mixing utterly distorts similarity).

# Offline ingest: batch embed + dual-index canary (pseudocode)
for batch in chunks.stream(size=256):         # large batch saturates GPU
    vecs = embed_offline(batch, model="v3")   # same model family as online
    new_index.upsert(batch.ids, vecs, meta={"model":"v3"})
# On model change: build v3 in background → validate recall → atomic cutover → drop v2
# Online query side:
@cache(ttl="1h")                              # cache hot query vectors
def embed_query(q): return embed_online(q, model=current_index_model())

④ Human-in-the-loop — fence high-risk actions behind an approval gate

One-line trade-off: synchronous blocking is simple but holds a connection/thread; async suspend scales but must persist the whole agent state.

[Principle] Models make mistakes, so irreversible high-risk actions (send email, transfer money, delete data, change prod config) can't be left to the agent's own call. The pattern is human-in-the-loop: when the agent reaches a high-risk tool, instead of executing it, it pauses, pushes "I intend to do X" to an approval queue, and continues only after approval. The hard part is that "wait for a human" may take hours or days—you can't have a thread/connection idle-waiting. So it must build on the durable execution of point ②: persist agent state, release all resources, and on the approval callback resume from the pause point.

# High-risk tool → suspend for approval (durable pseudocode)
def run_tool(call):
    if call.tool in HIGH_RISK:
        approval_id = approvals.create(call)        # push to approval queue
        decision = workflow.wait_for_signal(         # durable suspend, can wait days
            approval_id, timeout="3d")
        if decision != "approved":
            return refused(call)                     # don't execute if denied
    return execute(call)                             # execute only if approved/low-risk

Scaling & Optimization

• Semantic cache: reuse answers for similar queries (return on vector-neighbor hit); high-frequency FAQs skip the whole retrieve+generate chain, saving money and latency. Mind invalidation—knowledge-base updates must expire the cache.
• Retrieval sharding: 100M vectors won't fit one machine; shard by tenant/topic, route queries to relevant shards, avoid full-corpus scans (see Day 4 "Sharding").
• Multi-tier model routing: simple questions to a small model, complex tasks to a large one, routing by difficulty to cut cost (see Day 32 "LLM Serving").
• Incremental reindex: route doc updates through CDC, re-embed only changed chunks rather than rebuilding the whole corpus.
• Eval loop: maintain a fixed golden set of "question → expected source", run regression on every retrieval/model change, holding quality with recall/answer-correctness instead of tuning by vibes.

Common Pitfalls + Interview Follow-ups

• "Blame model hallucination when recall is the problem": 80% of bad RAG answers are failures to retrieve the right chunk, not the model making things up. Measure retrieval recall first, then talk generation.
• "Query and doc use different embedding models": similarity becomes noise, recall all wrong, and nothing alarms—the invisible killer noted above.
• "Agent state only in memory": any deploy/autoscale event restarts every running task from scratch, tokens burned for nothing. Long tasks must be persisted.
• "Agent infinite loop": no step cap, the model keeps calling tools without converging, blowing the budget. Need a hard cap + degrade to human.
• "Irreversible high-risk action executed automatically": a wrong email or deletion can't be rolled back. Irreversible actions always go human-in-the-loop.
• Follow-up: "Retrieval budget is only 300ms—what if hybrid+rerank can't finish?" → parallelize dense/sparse recall, rerank only top-N, use a distilled small cross-encoder, route hot queries to the semantic cache.

Deep-dive Resources

• Anthropic — Building Effective Agents: workflow vs agent, when to use an agent, human checkpoints, and other practices.
• Anthropic — Contextual Retrieval: contextual embedding + BM25 + rerank cuts retrieval failures by ~49%/67%.
• Lewis et al. 2020, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (NeurIPS): the original RAG paper.
• Designing Data-Intensive Applications (Kleppmann): the fundamentals of idempotency, state machines, and recoverable execution—the basis for durable agent execution.
• This series Day 31 (Hybrid Search & Reranking), Day 32 (LLM Serving), Day 39 (Workflow Engines)—this issue is where those three converge for AI products.

Going Deeper