Eval Beyond Benchmark

The Three Sins of Public Benchmarks: Contamination, Saturation, Construct Gap

Background & Principle

Three mechanisms decouple public scores from production reliability:

• Contamination: long-lived public sets like MMLU and HumanEval have almost certainly entered new models' pre/post-training data in some form. More insidiously, for SWE-bench — analyses of the Verified subset found roughly one-third of issues describe the fix directly in the issue text or comments, so models "copy" rather than "solve," inflating scores.
• Saturation: top models cluster in an 88%–92% ceiling band, so the benchmark loses discriminative power — score gaps fall into the noise and can't tell you whether A beats B in your setting.
• Construct Gap: a benchmark measures a proxy for the target, not the target. HumanEval tests "complete an isolated function," your job is "fix a cross-file bug in a 500k-line repo." Research shows that swapping SWE-bench's long GitHub issues for the terse, vague phrasing real users write drops relative success rates by 20–40%.

Underneath it all is Goodhart's law: when a metric becomes a target, it ceases to be a good metric. The whole field is overfitting public benchmarks, and the correlation between score and real capability decays year over year.

Hands-on

Turn "can I trust this benchmark" into a 30-second checklist pinned to your model-selection doc:

# See a shiny score? Ask these 4 first
1. How long public?      > 1 yr   → likely contaminated, discount it
2. Gap among top models? < 3pp    → it's noise, don't pick on this
3. Task shape like mine? single-fn/MCQ != multi-turn agent / long repo
4. Strict oracle?        weak tests pass "looks-right-but-wrong" answers

# Fails any one → that score can't drive a production decision

Build Your Eval Set Backward from Production Traces

Background & Principle

Hamel Husain hammers one point in Your AI Product Needs Evals: failed AI products almost always failed at having no evaluation system, yet a usable one starts shockingly small — a few dozen real examples plus one scorer is enough to run. The key isn't volume, it's distribution match: your eval inputs must look like production inputs, or a high score is just self-deception.

The method is error analysis / open coding: pull a batch of real traces, read outputs by hand, and label the failures (don't fix a rubric first — look at the data, then induce failure modes). The EvalGen paper (Shankar et al. 2024) names this phenomenon criteria drift — you need criteria to grade outputs, but grading outputs is exactly what helps you define the criteria. So an eval isn't a write-once static doc; it co-evolves with your data.

Hands-on

# Minimal pipeline: build an eval set backward from prod logs
import json, random

# 1) Stratified sampling: don't only grab successes; oversample failures
traces = load_prod_traces("2026-05")
hard = [t for t in traces if t.thumbs_down or t.retried or t.fallback]
sample = random.sample(traces, 60) + random.sample(hard, 40)  # deliberately skew to hard cases

# 2) Open coding: read + label failure modes (data first, rubric second)
#    e.g. wrong_tool / hallucinated_field / ignored_constraint / too_verbose

# 3) Freeze into a ground-truth eval set, commit to the repo
for t in sample:
    eval_set.append({"input": t.input, "context": t.context,
                     "expected": human_label(t),     # you label it, not the model
                     "slice": t.category})           # ← needed for §3 slicing
json.dump(eval_set, open("evals/v1.json", "w"))

Note the oversampling of hard cases in step 1: 95% of production is easy requests, so natural-distribution sampling dilutes the eval with easy examples and masks critical failures. The eval set should deliberately skew toward where you most fear being wrong.

Slice-Based Eval: Aggregate Scores Lie to You

Background & Principle

This is the most counterintuitive — and most valuable — rule in real-world eval. A "90% pass" agent might be 95% on "refund requests" and only 45% on "address-change requests" — and address changes happen to be your highest-volume or highest-risk scenario. The aggregate averages away this in-distribution catastrophe.

The fix is slice-based evaluation: cut along business-meaningful dimensions, report each slice's score separately, and watch the worst slice rather than the mean. Slice dimensions usually come from the failure modes you open-coded in §2, or business category, input length, language, user segment. It's also a natural defense against Goodhart — when you optimize the aggregate it's easy to sacrifice a small slice for the headline; slicing leaves that trade nowhere to hide.

Hands-on

from collections import defaultdict

def sliced_report(eval_set, run_agent, scorer):
    buckets = defaultdict(lambda: [0, 0])   # slice -> [pass, total]
    for ex in eval_set:
        ok = scorer(run_agent(ex), ex["expected"])
        b = buckets[ex["slice"]]
        b[0] += int(ok); b[1] += 1
    # key: print ascending by score, worst slice on top
    rows = sorted(((s, p/n, n) for s,(p,n) in buckets.items()),
                  key=lambda r: r[1])
    for s, acc, n in rows:
        flag = "  ⚠️ below bar" if acc < 0.8 else ""
        print(f"{s:<22} {acc:5.0%}  (n={n}){flag}")

# Example: aggregate 90%, but worst slice only 45% — that's what to fix
# change_address        45%  (n=20)  ⚠️ below bar
# multi_intent          63%  (n=15)  ⚠️ below bar
# refund                95%  (n=30)
# ── aggregate ──       90%

Your launch gate should hang on the worst slice ("every slice ≥ 80% to ship"), not the aggregate mean. A 45% slice masked by the aggregate is exactly where post-launch complaints come from.

Agentic Eval: Evaluate the Trajectory, Not the Answer

Background & Principle

Traditional benchmarks ask once and check once; a real agent is a multi-turn, tool-calling trajectory interacting back-and-forth with a (possibly misleading) user. Two paradigm shifts:

• From outcome to trajectory: only checking whether the final answer is right lets through fragile trajectories that "got lucky but called the wrong tools in the wrong order." Evaluate the intermediate steps — which tools were called, were policies violated, were there redundant actions.
• From pass@1 to pass^k (consistency): τ-bench (Yao et al. 2024) proposes pass^k — the probability that the same task succeeds on all of k consecutive runs. In the paper, SOTA function-calling agents already fall below 50% single-task success, and in the retail domain pass^8 drops below 25%. That gap is the chasm between "the demo works" and "production is reliable." Its scoring is hardcore too: it ignores conversational phrasing and compares the final database state against the annotated goal state.

For individuals and small teams, two ideas from τ-bench are enough: state-based scoring (verify side effects, not phrasing) + pass^k consistency (run the same task several times to see if it's stable).

Hands-on

def pass_hat_k(task, run_agent, check_final_state, k=8):
    # pass^k: stable only if all k runs pass; any failure → False
    return all(check_final_state(run_agent(task)) for _ in range(k))

def check_final_state(result):
    # state-based: verify side effects, ignore how it phrased things
    return (db.get("order#123").address == "new addr"
            and not charged_extra_fee())     # also verify "did no harm"

# Trajectory compliance: is the PROCESS right, not just the result
def check_trajectory(trace):
    if "refund" in trace.tools_called and not trace.asked_confirmation:
        return "VIOLATION: refunded without confirmation"  # policy breach
    return "ok"

For an agent, "succeeded once" is meaningless — pass^8 is the launch signal. An agent with pass@1=80% but pass^8=30% means, on average, two out of every three users will hit a failure on some turn.