Automatic Prompt Optimization

Hand-tuning vs Auto-optimization: Ask if It's Worth It First

Background & Principle

Auto-optimization is fundamentally black-box search using eval scores as a gradient: it climbs only toward the metric you define. So the coarser the metric, the more crooked the result. Three conditions must all hold to trigger it: (1) an objective or semi-objective metric (accuracy, F1, format compliance, or a calibrated LLM-judge); (2) dozens to hundreds of samples with ground truth, splittable into independent train / val / test; (3) the prompt is called at high frequency, justifying a one-time optimization cost. Conversely, one-off tasks, open-ended creation with no scorable metric, or single-digit sample counts—hand-tuning is faster and cheaper. One often-ignored criterion: is the task stable. If requirements shift weekly and the schema changes daily, optimized artifacts expire fast and maintenance overwhelms the gain.

DSPy: Compile Prompts as Programs

Background & Principle

The core shift in DSPy (Khattab et al. 2023, arXiv 2310.03714): you declare the input/output signature and module structure, and hand "what the actual prompt looks like" to an optimizer to search. A Signature defines fields (e.g. text -> label), a Module (e.g. ChainOfThought) defines how it's called, and the optimizer jointly optimizes two things on the trainset: each module's instruction wording, and automatically bootstrapped few-shot examples. MIPROv2 (Opsahl-Ong et al. 2024, arXiv 2406.11695) uses Bayesian optimization to search instruction+demo combinations; it does credit assignment for multi-stage LM programs without module-level labels—you give only the final metric and it apportions across modules. The value: when you switch models or change requirements, you re-compile once instead of rewriting every prompt.

Example

import dspy
dspy.configure(lm=dspy.LM("anthropic/claude-haiku-4-5"))

class Classify(dspy.Signature):
    """Judge the urgency of a support ticket."""
    ticket: str = dspy.InputField()
    urgency: str = dspy.OutputField(desc="low|medium|high")

program = dspy.ChainOfThought(Classify)

# metric: just return a comparable score
def metric(gold, pred, trace=None):
    return gold.urgency == pred.urgency

from dspy.teleprompt import MIPROv2
opt = MIPROv2(metric=metric, auto="medium")      # search intensity tier
compiled = opt.compile(program, trainset=train, valset=val)
compiled.save("classify.v3.json")                  # artifact into registry (Day 35)

Key mental model: you're not tuning "prompt text," you're tuning three knobs—metric, trainset, optimizer tier. The optimized instruction often reads unremarkable or even odd, yet scores higher on val—which is exactly the point of ceding aesthetic judgment to data.

Instruction Search: The Real Gains of APE and OPRO

Background & Principle

Two classic routes. APE (Zhou et al. 2022, arXiv 2211.01910) treats the "instruction" as a program: have an LLM generate a batch of candidate instructions, then score and select the best via the target model's eval—matching or beating human-written on 24 NLP tasks. OPRO (Yang et al. 2023, arXiv 2309.03409) goes further: it feeds the "history of (prompt, score) pairs" sorted by score back to the LLM, having it propose a better next version based on the trajectory, forming an iterative climb. Prompts found this way are often surprising—OPRO's famous find on GSM8K is "Take a deep breath and work on this problem step by step," which outscored human-written CoT cues. This shows two things: the objective is score, not readability; and this win depends strongly on the specific model being optimized—switch models and you usually re-search.

Example

You can hand-roll a minimal OPRO loop without any third-party library:

def opro_step(history, n=8):
    # history: [(instr, score), ...] sorted ascending by score
    shown = "\n".join(f"instr: {i}\nscore: {s:.2f}" for i,s in history[-10:])
    meta = f"""Below are instructions and their scores (higher is better).
{shown}
Write {n} new instructions aiming for a higher score. Output instructions only."""
    cands = propose(meta)                     # LLM generates candidates
    return [(c, eval_on_valset(c)) for c in cands]  # score each
# iterate, merge new (instr,score) into history, pin on test after convergence

Engineering rule: candidates must be scored on held-out val, and the final version must be verified on a test set that never participated in the search—otherwise what you found is a "score-pumping incantation," not a real improvement. For a shortcut, Anthropic Console's prompt improver is a zero-code entry point—good for getting a strong baseline before considering search.

Auto Few-shot Selection and Overfitting Governance

Background & Principle

An underrated fact: on many tasks, auto-bootstrapped few-shot examples yield more than fine-tuning instruction wording. DSPy's BootstrapFewShot does exactly this: a teacher model runs on the trainset, the metric filters for "complete trajectories that got it right" as examples, then packs them into the prompt. This is more systematic than hand-picking. But examples are double-edged: selected samples carry their distribution, format, and even biases into the prompt, which can backfire on out-of-distribution inputs. So auto-optimization must borrow ML discipline: train / val / test three-way split, search touches only train+val, reporting trusts only test; and fold the artifact into Day 35's prompt registry—versioned, rollback-able, recording "which model, which data, which metric it was compiled with." When you switch models or data drifts, the registry triggers re-compilation instead of letting production silently degrade.

Example

# Data splitting is the seatbelt of auto-optimization, not optional
train, val, test = split(data, 0.6, 0.2, 0.2)

from dspy.teleprompt import BootstrapFewShot
opt = BootstrapFewShot(metric=metric,
        max_bootstrapped_demos=4,   # examples generated by teacher
        max_labeled_demos=4)      # examples taken directly from train
compiled = opt.compile(program, trainset=train)

# trust only the test score—it never participated in search
print(evaluate(compiled, test))
# record provenance: model + data version + metric + score → registry

Governance checklist: the gap between val and test scores is your overfitting thermometer—a large gap means you over-searched, so dial back search intensity or add data. After launch, monitor production score vs test score; when prompt drift appears (same prompt shifting in quality due to a model update), that provenance in the registry lets you reproduce and re-compile with one click.