Context Engineering

Lost-in-the-Middle: The Real Effective Length of Long Context

Background & Principles

Liu et al. 2023's Lost in the Middle: How Language Models Use Long Contexts (TACL 2024) is the foundational reference. They had models find an answer across a set of documents, placing the ground-truth document at positions 1, 5, 10, 15, 20 and plotting accuracy. Every tested model (GPT-3.5, Claude, Longchat) showed the same U-curve: ~75% accuracy at the head and tail, collapsing below 50% in the middle — worse than turning retrieval off entirely.

Two reasons. First, in pretraining, token dependencies are mostly local or anchored to document head (causal masking biases attention toward recent tokens; RoPE-style positional encodings have long-tail decay that weakens far-range attention). Second, post-training instruction data is mostly "short prompt → answer"; the model has seen few samples of "instruction at the tail, key facts in the middle".

In 2024 NoLiMa (Modarressi et al.) went further: when question and answer share no lexical overlap, supposedly 100K+ models fall below baseline at 32K. Chroma's 2025 Context Rot report retested 18 frontier models (Claude Sonnet 4, GPT-4.1, Gemini 2.5) and concluded the same: effective context is much shorter than advertised max context, and the gap grows with task difficulty.

Hands-on Example

Run a minimal NIAH (Needle in a Haystack) test to measure the real effective length of your go-to model:

import anthropic, random, string

client = anthropic.Anthropic()
needle = "BigCat's meeting password is PURPLE-OWL-9182."

def haystack(n_tokens, needle_pos):
    # Pad to n_tokens with Paul Graham essays
    filler = open("pg_essays.txt").read()
    chunks = filler.split("\n\n")
    random.shuffle(chunks)
    text = "\n\n".join(chunks)[:n_tokens * 4]  # ~4 char/token approximation
    pos = int(len(text) * needle_pos)
    return text[:pos] + "\n\n" + needle + "\n\n" + text[pos:]

for length in [8000, 32000, 100000, 180000]:
    for pos in [0.1, 0.5, 0.9]:
        ctx = haystack(length, pos)
        r = client.messages.create(
            model="claude-sonnet-4-6",
            max_tokens=128,
            messages=[{"role":"user","content": ctx +
                "\n\nQ: What is BigCat's meeting password? Answer with the password only."}])
        print(length, pos, r.content[0].text.strip())

You'll end up with a (length × position) hit table. Trust only the cells with ≥ 95% hit rate — beyond that, regardless of the marketed K, don't put critical information there.

Information Layout Principles: U-Shape + Anchor + Restate

Background & Principles

Since head and tail carry the highest attention weight, your prompt's physical layout should mirror that importance gradient. Anthropic's official long-context guidance (claude.com/docs · Long context tips) is blunt: put long reference documents at the very top of the prompt and the question at the end of the user turn. This has been measured repeatedly on Claude — long-doc Q&A accuracy goes up 10–20%.

Three concrete moves:

• Anchor: state core constraints and the current goal twice — once in the system block, once at the end of the user turn. Anthropic's long-doc example explicitly places the question after <documents>, not before.
• Restate: when Claude must cite a long doc, ask it to "quote the relevant sentences verbatim" before answering. This forces the model to pull the information from the middle back into the active attention window before generating.
• Stable prefix: put unchanging instructions / schema / examples at the very top to maximize KV cache hit rate (the same idea as Day 1's prefix caching).

Hands-on Example

<documents>
  <document index="1">
    <source>contracts/acme-2024.pdf</source>
    <content>...8K tokens...</content>
  </document>
  <document index="2">...</document>
</documents>

Instructions (restated):
- Cite only clauses explicitly written in the documents above. Do not extrapolate.
- First list the verbatim passages you will cite, then give the conclusion.

Question:
Can we unilaterally terminate the Acme contract at month 18? Walk through the basis clause by clause.

RAG's Real Bottleneck: Chunk Design Matters Way More Than the Embedding Model

Background & Principles

Standard team workflow when shipping RAG: fixed 512-token chunk + 50 overlap + OpenAI embedding + simple top-k. Quality is bad, so they swap embedding models, then swap vector DBs. Same outcome. Why:

• Semantic boundaries get butchered: fixed-size chunking cuts inside sentences and paragraphs. A "However, the above does not apply when…" pivot gets split across two chunks; retrieving one of them gives a confidently wrong answer.
• Contextless chunks: a chunk reads "the clause becomes effective within 30 days" — but which clause? The model has no idea. Anthropic's 2024 Contextual Retrieval blog showed that prepending each chunk with "this chunk in the context of the document" cuts retrieval failures by 49%; adding BM25 brings it to 67%.
• Single granularity: legal clauses want 200 tokens; source code wants per-function; equity research wants per-section. One fixed size is one hammer for every nail.

The three things that actually determine your RAG ceiling: chunk boundaries (semantic), chunk context (contextual prefix), and retrieval fusion (hybrid + rerank). Embedding model choice, among reasonable options, is a ±5% knob.

Hands-on Example

Anthropic's recommended Contextual Retrieval flow (drop-in for your own corpus):

# Step 1: generate context for each chunk (batch with Claude Haiku + prompt caching)
context_prompt = """
<document>{whole_doc}</document>
Here is the chunk we want to situate within the whole document:
<chunk>{chunk}</chunk>
Please give a short succinct context to situate this chunk within
the overall document for improving search retrieval. Answer only
with the context, nothing else.
"""

# Step 2: prepend context to chunk, then build embedding + BM25 indexes
indexed_chunk = f"{generated_context}\n\n{original_chunk}"

# Step 3: at query time, hybrid (dense + BM25), top-150 → rerank to top-20
dense_hits = vector_store.search(query, k=150)
bm25_hits  = bm25_index.search(query, k=150)
fused = reciprocal_rank_fusion([dense_hits, bm25_hits])
top20 = cohere_rerank(query, fused[:150], top_n=20)

Critical engineering detail: in step 1, the {whole_doc} sits in the same cached system + document prefix across all chunks of that document; with prompt caching enabled, cost drops to about 1/10. Anthropic's own number: contextualizing a 100-page document costs about $1.02 — cheaper than swapping embedding models, and significantly more effective.

Context Compaction: Controlled Compression of Long Conversations

Background & Principles

Agents and long chats often see accuracy fall off a cliff around turn 50. The cause isn't always token overflow — often you're still well under max context, but the model is already confused. This is context rot: relevant signal diluted by irrelevant history, combined with lost-in-the-middle, pushes attention onto irrelevant tokens.

Four mainstream strategies for long history; the engineering choice maps to different trade-offs:

• Sliding window: keep the most recent N turns. Simplest. Drops early constraints (you said "don't use jQuery" 3 turns ago — gone).
• Summarization compaction: every K turns, compress older turns into a structured summary (user preferences, completed steps, open questions). Claude Code's built-in auto-compaction is exactly this: keep the most recent few turns verbatim plus a structured summary of everything before.
• Hierarchical memory: working memory (recent) + episodic memory (event summaries) + semantic memory (vector retrieval). MemGPT (Packer et al. 2023, arxiv.org/abs/2310.08560) is the canonical implementation and Lilian Weng's recommended template in LLM Powered Autonomous Agents.
• Selective pruning (relevance-based): score every historical turn with a small model and drop low scores wholesale, instead of by time. Works well for tool-use-heavy agents — most tool outputs are used once and never referenced again.

Hands-on Example

Minimal compaction loop for a long-running coding agent:

def maybe_compact(history, model="claude-sonnet-4-6", threshold=120_000):
    tok = count_tokens(history)
    if tok < threshold:
        return history
    # Keep the last 6 turns verbatim; compress everything before into structured summary
    recent, old = history[-6:], history[:-6]
    summary = client.messages.create(model=model, max_tokens=2000,
        system="You compress agent histories. Output strict JSON.",
        messages=[{"role":"user","content": f"""
Compress the following turns into JSON with keys:
- goal: the user's overall objective
- decisions: list of decisions made (with rationale)
- artifacts: files/code created or modified (paths only)
- open_questions: anything explicitly deferred
- forbidden: constraints user said NOT to do
<turns>{serialize(old)}</turns>
"""}]).content[0].text
    return [{"role":"system","content":f"<prior_session>{summary}</prior_session>"}] + recent

Key lesson: the compression schema matters more than the compression algorithm. Forcing the model to emit fixed keys (goal / decisions / artifacts / open_questions / forbidden) lets the next turn directly reference <prior_session>.forbidden — far more robust than free-form summarization.