RAG in Production

Chunking: fixed 512 is a beginner trap; structure-aware is the production config

Background & principle

The root problem with fixed-size chunking: semantic boundaries do not coincide with character boundaries. A 512-token cut may put "why" in chunk A and "because…" in chunk B — their embeddings can't see each other, retrieval will never recover both. This is not a chunk-size tuning problem; it's a strategy problem.

Four generations of chunking: (1) fixed size — naive, known to fail; (2) recursive character splitter (the LangChain classic) — backs off paragraph → sentence → word, the minimum entry bar to anything usable; (3) structure-aware — splits by markdown headers, code AST, PDF sections, preserving semantic units; (4) parent-child / hierarchical — small chunks for retrieval, larger chunks (parent paragraph or full section) for the LLM, eliminating "precise recall but mutilated context."

Anthropic's 2024 Contextual Retrieval is the fifth generation: before embedding each chunk, Claude writes a 1-2 sentence prefix locating it within the document. The same RAG pipeline, with only that one change, drops failure rate by 35%. Why: embedding models produce a context-less vector for an isolated chunk; an explicit 50-100 token prefix snaps the embedding into the correct semantic neighborhood. Combined with prompt caching, contextualizing a 1M-token corpus costs about $1.02 — making this the highest-ROI single change in 2024 RAG engineering.

Hands-on example

Structure-aware + parent-child + contextual prefix (minimum working version):

from langchain.text_splitter import MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter
import anthropic

# —— 1. Structure-aware: split on markdown headers first (= parent chunks) ——
header_splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=[("#", "h1"), ("##", "h2"), ("###", "h3")]
)
parent_chunks = header_splitter.split_text(doc)  # ~1500 tokens each

# —— 2. Parent-child: cut each parent into retrieval-sized children ——
child_splitter = RecursiveCharacterTextSplitter(
    chunk_size=400, chunk_overlap=50,
    separators=["\n\n", "\n", ". ", " "]
)

# —— 3. Contextual prefix: have Claude add a context tag per child ——
# Put the full doc behind cache_control=ephemeral; pay input cost once for 1M tokens
client = anthropic.Anthropic()

def contextualize(full_doc, chunk):
    msg = client.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=120,
        messages=[{"role":"user", "content":[
          {"type":"text",
           "text": f"<document>{full_doc}</document>",
           "cache_control":{"type":"ephemeral"}},  # ← key
          {"type":"text",
           "text": f"<chunk>{chunk}</chunk>\nGive a 1-2 sentence "
                   "context placing this chunk in the doc. Just the context."}
        ]}
    )
    return msg.content[0].text

# Indexing: embed(contextual_prefix + child_chunk)
# Serving: retrieve child → fetch its parent → hand parent to LLM for richer context
indexed = []
for parent in parent_chunks:
    for child in child_splitter.split_text(parent.page_content):
        prefix = contextualize(doc, child)
        indexed.append({
          "embed_text": prefix + "\n" + child,  # used for embedding / BM25
          "parent_text": parent.page_content,    # fed to the LLM
          "metadata": parent.metadata
        })

Hybrid Search: BM25 isn't a relic — it's the half dense embedding will never catch

Background & principle

The "dense embedding has obsoleted BM25" narrative since 2020 is a benchmark mirage. Real life: a user types "GPT-4o vs Claude 4.7" — dense embedding retrieves every "LLM comparison" article; BM25 retrieves docs that literally contain those two tokens. "connection refused on port 5432," "erro_user_42," "RFC 9457" — these are BM25 territory, where dense gets crushed. The reason: embedding projects tokens into a continuous semantic space, lossy-compressing the exact-match signal of rare tokens; BM25 preserves token-level precision.

Anthropic's Contextual Retrieval ablation is unusually clean: dense-only failure rate 5.7%, adding BM25 (Contextual BM25 + Contextual Embeddings) drops it to 2.9% — nearly halved. Azure AI Search, Pinecone, Weaviate, Elasticsearch 8.x all ship hybrid by default. This is no coincidence; it's the de facto standard of 2024 production RAG.

For fusion, pick RRF (Cormack et al. SIGIR 2009): no need to normalize scores across sides, only ranks. score(d) = Σ 1/(k + rank_i(d)), k typically 60. Far more robust than weighted sums — weighted sums force endless tuning; RRF is immune to score scale differences. Elastic defaults to k=60, Anthropic's report uses 60.

Hands-on example

Ten lines of RRF on top of dense + BM25 (using bm25s + any vector store):

import bm25s
from collections import defaultdict

# —— Indexing ——
corpus_texts = [item["embed_text"] for item in indexed]
bm25 = bm25s.BM25()
bm25.index(bm25s.tokenize(corpus_texts, stopwords="en"))
# dense_index = your_vector_store.upsert(...)

# —— Query time ——
def hybrid_retrieve(query, top_k=50, k=60):
    # Each side returns top_k
    bm25_results = bm25.retrieve(bm25s.tokenize([query]), k=top_k)
    dense_results = dense_index.query(embed(query), top_k=top_k)

    # —— RRF fusion ——
    scores = defaultdict(float)
    for rank, doc_id in enumerate(bm25_results.ids[0]):
        scores[doc_id] += 1.0 / (k + rank + 1)
    for rank, hit in enumerate(dense_results.matches):
        scores[hit.id] += 1.0 / (k + rank + 1)

    fused = sorted(scores.items(), key=lambda x: -x[1])
    return [doc_id for doc_id, _ in fused[:top_k]]

# Key: each side returns 50; RRF keeps top 50.
# Don't truncate to top 5 here — leave the precision work to the reranker.

Reranker: cross-encoders shrinking top-50 to top-5 — RAG's best-priced layer

Background & principle

Bi-encoder vs cross-encoder is a fundamental architectural split. Bi-encoder (OpenAI text-embedding-3, Cohere embed, BGE): query and doc become vectors independently. Indexable offline, millisecond queries — but the model never sees the pair together, so subtle relevance signals are lost. Cross-encoder (Cohere Rerank, BGE-reranker-v2-m3, ms-marco-MiniLM): query and doc are concatenated as [CLS] query [SEP] doc and passed jointly through a BERT-like model, emitting a relevance score directly. Precision approaches LLM-as-judge using only 80M-300M parameters.

The cross-encoder catch: compute scales linearly with the candidate count. Reranking top 50-100 is fine (50ms-300ms). Reranking 100K docs as first-stage retrieval is hundreds of seconds. The correct architecture is always two-stage: bi-encoder/BM25 retrieves a large candidate set (top 50-200), then the cross-encoder reranks to top 5-10.

The ROI numbers. BEIR benchmark: dense baseline NDCG@10 = 0.42; add ms-marco-MiniLM-L-6-v2 reranker = 0.54 (+28%). Anthropic's Contextual Retrieval report: on top of hybrid + contextual, adding Cohere Rerank drops failure rate from 2.9% to 1.9% (another -35%). Cohere's own BEIR reproductions show rerank-v3 lifting NDCG by 12-18 points on top of hybrid. Cost: +50-200ms per query and ~$1 per thousand queries (Cohere pricing) or zero cash with self-hosted BGE. This is the single highest-ROI component in RAG. No close second.

Hands-on example

BGE-reranker (open source, self-hosted) primary + Cohere (managed) fallback — a stable production config:

from FlagEmbedding import FlagReranker
import cohere

# —— Default: self-hosted BGE-reranker-v2-m3 (multilingual, 300M params, fits in 8GB) ——
# Open source, zero cash cost, p95 ~80ms on an A10
reranker_local = FlagReranker("BAAI/bge-reranker-v2-m3", use_fp16=True)

# —— Fallback: Cohere Rerank-v3 (managed, stable, no GPU needed) ——
co = cohere.Client(API_KEY)

def rerank(query, candidates, top_n=8):
    # candidates is the top-50 hybrid output (list of {id, text})
    try:
        pairs = [(query, c["text"]) for c in candidates]
        scores = reranker_local.compute_score(pairs, normalize=True)
    except Exception:
        result = co.rerank(
            model="rerank-v3.5",
            query=query,
            documents=[c["text"] for c in candidates],
            top_n=top_n
        )
        return [candidates[r.index] for r in result.results]

    ranked = sorted(zip(candidates, scores), key=lambda x: -x[1])
    return [c for c, _ in ranked[:top_n]]

# —— Key: the post-rerank top_n is what actually goes to the LLM. ——
# Don't dump 50 chunks on the model and expect it to "pick the best."
# That wastes tokens and triggers lost-in-the-middle.
top_chunks = rerank(query, hybrid_candidates, top_n=8)

Query Transformation: the user's query is the weakest link in RAG

Background & principle

The core mismatch: chunks in the corpus are in "answer form" (declarative, long context); user queries are in "question form" (short, pronoun-heavy, missing context). In embedding space, the two are naturally far apart. That's why your demo "works" with carefully phrased queries but breaks on real user input — the model isn't worse, the query just isn't in the semantic neighborhood of the answer.

Three mainstream query transformations:

• HyDE (Hypothetical Document Embeddings, Gao et al. ACL 2023): have the LLM pretend to answer the query, then use the embedding of the fake answer for retrieval. Intuition: even a poor fake answer is closer to the true answer in semantic space than the question is. BEIR shows +6-15 NDCG points zero-shot.
• Multi-query (LangChain / RAG-Fusion): have the LLM rewrite the query into 3-5 variants (synonyms, different angles, more specific), retrieve each, merge via RRF. Covers paraphrase blind spots.
• Step-back prompting (Zheng et al. ICLR 2024, Google DeepMind): generate a more abstract / higher-level version of the question, retrieve principle-level answers, then combine that with the original query in the final prompt. Boosts reasoning-heavy QA by 7-27%.

These three are not mutually exclusive. Anthropic's 2024 engineering note: HyDE + multi-query stack best on semantically vague queries; step-back gives the largest lift on "understand the principle first, then apply it" domains (math, law, medicine). But cost stacks too — each query adds 1-2 LLM calls, +500ms-1s latency. So in practice, transformations should be routed by query type: short specific ID-style queries skip everything; long natural-language queries get the full stack.

Hands-on example

HyDE + multi-query combo with query-type routing:

import anthropic, re
client = anthropic.Anthropic()

# —— 1. HyDE: generate a fake answer, embed that instead of the query ——
def hyde(query):
    msg = client.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=200,
        messages=[{"role":"user", "content":
          f"Write a concise, factual 3-4 sentence answer to this question. "
          f"If you don't know, write what such an answer would likely contain.\n\n{query}"}]
    )
    return msg.content[0].text  # fake answer; embed THIS, not the query

# —— 2. Multi-query: rewrite into 3 variants ——
def multi_query(query):
    msg = client.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=300,
        messages=[{"role":"user", "content":
          f"Rewrite the following query as 3 distinct search queries. "
          f"One per line, no numbering, no quotes.\n\nQuery: {query}"}]
    )
    return [q.strip() for q in msg.content[0].text.strip().split("\n") if q.strip()]

# —— 3. Router: pick transformation by query shape ——
def route_and_retrieve(query):
    # Short, or contains ID/version/error code → skip transformation, let BM25 carry it
    if len(query.split()) < 5 or re.search(r'\b[A-Z]{2,}[-_]?\d+\b|v\d+\.\d+', query):
        return hybrid_retrieve(query, top_k=50)

    # Long natural-language query → full HyDE + multi-query
    queries = [query] + multi_query(query)
    candidates = defaultdict(float)
    for i, q in enumerate(queries):
        # Dense uses HyDE fake answer; BM25 keeps the original (don't rewrite keywords)
        hyp = hyde(q)
        for rank, doc_id in enumerate(hybrid_retrieve_split(
              dense_query=hyp, bm25_query=q, top_k=50)):
            candidates[doc_id] += 1.0 / (60 + rank + 1)  # RRF
    return sorted(candidates, key=candidates.get, reverse=True)[:50]