AI Observability

Trace / Span: An Agent's Debug Unit Isn't a Log Line

Background

An agent call chain is nested: a top-level user request (trace) wraps multiple LLM calls, each wrapping tool calls, each possibly wrapping sub-agents. Recording this with flat log lines flattens a tree into a column — you can never reconstruct "which tool's output caused the wrong decision next turn." Distributed tracing's span model fits natively: each operation is a span (with start/end/attributes), spans link by parent_id into a tree, and the whole tree is a trace.

Since 2024 OpenTelemetry's GenAI semantic conventions standardized this: they specify which attributes an LLM call records (model, input/output tokens, prompt, tool calls, provider), so a LangChain agent's span looks identical to a raw OpenAI call — comparable across frameworks and vendors. So don't invent your own field names; align with the gen_ai.* conventions. Langfuse and Arize Phoenix are both built on OTel.

Example

Add spans to a homemade harness with the OTel SDK, aligning attribute names to the GenAI conventions:

from opentelemetry import trace
tracer = trace.get_tracer("my-agent")

def llm_call(msgs, tools):
    with tracer.start_as_current_span("llm.generate") as sp:
        r = client.messages.create(model=MODEL, messages=msgs, tools=tools, max_tokens=2048)
        # align to OTel gen_ai.*, don't invent fields
        sp.set_attribute("gen_ai.request.model", MODEL)
        sp.set_attribute("gen_ai.usage.input_tokens",  r.usage.input_tokens)
        sp.set_attribute("gen_ai.usage.output_tokens", r.usage.output_tokens)
        sp.set_attribute("gen_ai.response.finish_reason", r.stop_reason)
        sp.set_attribute("app.prompt_version", PROMPT_VERSION)  # custom: which prompt
        return r

The key is that span nesting must follow the agent's logical structure: is a tool call a sibling or a child of the llm span? Should a sub-agent's whole tree link back to the parent trace? Get these parent relationships right and the UI shows at a glance "7K of the 11.2K tokens went to a repeated search on turn three."

Cost & Token Attribution: Every Span Keeps a Ledger

Background

LLM cost isn't uniform like CPU — it concentrates in a few expensive paths: a feature whose prompt missed cache, a query class that triggers an 8-turn agent loop, a long doc re-stuffed into context every time. Aggregate billing hides these; you need to attribute cost down the span tree onto dimensions: which feature, which user, which prompt version, which step. The span attributes from §1 feed this directly — each LLM span already records input/output tokens and model; add cache_read tokens plus business dimensions and you can slice cost any way you like.

Three derived metrics to watch most: cache hit rate (higher prefix-caching hits = more saved on repeated prefixes — see Day 15/16), turns per task (a spike usually means degraded tool descriptions or the model going in circles), and p95 of per-task cost (averages get dragged down by cheap requests; the long tail is what burns money).

Example

Record full cost fields on the span so the backend can aggregate by dimension:

# record enough to reconstruct cost (incl. cache hits)
u = r.usage
sp.set_attribute("gen_ai.usage.input_tokens",  u.input_tokens)
sp.set_attribute("gen_ai.usage.output_tokens", u.output_tokens)
sp.set_attribute("gen_ai.usage.cache_read_tokens",  getattr(u, "cache_read_input_tokens", 0))
sp.set_attribute("gen_ai.usage.cache_write_tokens", getattr(u, "cache_creation_input_tokens", 0))
# business dims: the group-by keys for attribution
sp.set_attribute("app.feature", "refund_agent")
sp.set_attribute("app.user_tier", user.tier)

# backend query (pseudo-SQL): find the money sinks
# SELECT app.feature, SUM(cost), AVG(turns), PERCENTILE(cost, 0.95)
# FROM spans WHERE day = today GROUP BY app.feature ORDER BY 2 DESC

With that table, optimization has targets: low-cache-hit features get a fixed prompt prefix; high-turn features get their tool descriptions checked; p95-tail features get context trimming or model routing. Without attribution, all of this is guesswork.

Drift Detection: Did the Input Change, or the Model?

Background

Offline eval is a one-time snapshot: test before release, ship if it passes. But production is alive and drifts in two directions:

• Input / data drift: phrasing, topics, language distribution shift. Your prompt and few-shots were tuned for the old distribution; accuracy drops on the new one, but the inputs themselves throw no error. Detection borrows from classic ML: embed production inputs, compare to a reference period, compute PSI (Population Stability Index) or KS / Jensen-Shannon divergence. Evidently's rule of thumb: PSI > 0.25 means significant drift.
• Output / behavior drift: input distribution is stable but output changed — the most insidious kind, usually from a silent provider model update, or you changing the prompt/temperature. Monitor structural features of outputs (length, format-compliance rate, refusal rate, tool-selection distribution, JSON parse-failure rate); any sudden shift is a signal.

Keep the two axes separate: input drift tells you "time to add few-shots / re-test"; output drift while input is stable almost always means the model or config changed — the most dangerous failure mode of a hosted API you can't pin.

Example

import numpy as np
def psi(ref, cur, bins=10):
    # ref/cur: a scalar distribution for reference vs current period
    # (e.g. an embedding component, or answer length)
    q = np.quantile(ref, np.linspace(0, 1, bins+1))
    q[0], q[-1] = -np.inf, np.inf
    r = np.histogram(ref, q)[0] / len(ref) + 1e-6
    c = np.histogram(cur, q)[0] / len(cur) + 1e-6
    return float(np.sum((c - r) * np.log(c / r)))

# daily compare: embedding projection / answer length / refusal rate
score = psi(ref_lengths, today_lengths)
if score > 0.25:                       # Evidently rule of thumb
    alert(f"output length drift PSI={score:.2f}, check model version / prompt change")

The cheapest practical "canary": maintain a set of fixed probe prompts (20–50 with inputs that never change), run them on a daily schedule, log the outputs. Inputs are constant, so any output change points straight at the model or config — the cheapest sentinel for detecting silent provider updates.

Online Eval: Scoring Real Traffic Continuously

Background

Day 6 / Day 29 covered offline eval: golden set + LLM-as-judge, run before release. The problem: a golden set always lags the real distribution and can't cover production's long-tail cases. Online evaluation fills that gap: sample production traces, score them with evaluators in (near-)real-time, and write the scores back as a metric you can alert on, dashboard, and slice. Arize Phoenix calls this online evals — attach an evaluator to incoming traces so each live execution gets scored as it runs.

Three evaluator types, stacked by increasing cost and decreasing coverage:

• Rules / heuristics (cheapest, full traffic): is the JSON parseable, are required fields present, did it hit a refusal template, is the length anomalous. Sub-millisecond, run on 100% of traffic.
• LLM-as-judge (expensive, sampled): relevance, faithfulness (is the answer supported by the retrieved evidence), is it off-topic. Run only on a sampled X% of traffic to control cost.
• Human labels / user feedback (most expensive, sparsest): 👍/👎, manual spot review. Serves as the calibration baseline for the LLM judge — the judge itself drifts and must be re-aligned with humans periodically.

The key engineering constraint: online eval must be async and off the hot path — never in the user's response critical path. Scoring runs in a background worker after the trace lands; it can be slow, but it must not slow the product.

Example

# off-path worker: consume from trace queue, tier-score, write metrics
def score_trace(trace):
    out = trace.final_output
    # 1) rule tier: full traffic, sub-ms
    emit_metric("eval.json_valid", is_valid_json(out), trace.id)
    emit_metric("eval.refused",   hit_refusal(out),    trace.id)
    # 2) LLM-judge tier: sample 5% to control cost
    if sample(0.05):
        s = judge(question=trace.input, answer=out,
                  context=trace.retrieved_docs,
                  rubric="Is the answer fully supported by context? 1-5")
        emit_metric("eval.faithfulness", s, trace.id)
        if s <= 2: alert(f"low-faithfulness answer trace={trace.id}")

Put the p50 of eval.faithfulness on a time-series dashboard; a drop is the earliest signal of a quality regression — days before users complain. Overlay the drift axes from §3: if faithfulness drops while the input distribution is stable, the finger points at a model or retrieval change.