Personal AI Infra

LLM Gateway: Funnel Every Call Through One Endpoint

Background & Principle

The gateway is the single entry point to the whole stack. Apps no longer know about "Anthropic," "OpenAI," or "Gemini"—they only know a local endpoint http://localhost:4000/v1 speaking the OpenAI ChatCompletions dialect. Behind it, the gateway handles the real provider adaptation, auth, accounting, caching, logging. LiteLLM Proxy is the lowest-friction choice for individuals: one config.yaml maps 100+ providers to unified model names, and it runs as an OpenAI-compatible server.

The key payoff is decoupling: switching the underlying model is a one-line change in the gateway, with zero downstream app changes; capping an app, rotating a key, or adding a fallback all happens at the hub. That's exactly what the diagram below shows—all four layers hang behind the gateway, and the app side sees only one endpoint.

Hands-on

# config.yaml — map unified model names to real providers
model_list:
  - model_name: smart            # apps only say "smart"
    litellm_params:
      model: anthropic/claude-opus-4-8
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: cheap
    litellm_params:
      model: anthropic/claude-haiku-4-5
      api_key: os.environ/ANTHROPIC_API_KEY

# Start: litellm --config config.yaml   →  localhost:4000
# App side is provider-agnostic:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:4000/v1", api_key="sk-my-virtual-key")
client.chat.completions.create(model="smart", messages=[...])  # call Claude via the OpenAI SDK

Cross-Model Routing & Fallback: Pick by Task, Back Up by Availability

Background & Principle

Axis 1 (cost routing): not every call deserves a frontier model. Classification, extraction, formatting, the routing decision itself—Haiku-tier is plenty, an order of magnitude cheaper. Flipping the default from "expensive model" to "cheap by default, expensive on demand" is the fastest way to halve an individual's token bill.

Axis 2 (availability fallback): providers rate-limit, 5xx, and occasionally refuse. A fallback is an ordered model chain: if the primary errors, auto-try the next. OpenRouter exposes this as a models array—listed by priority, an error on one falls through to the next, and you're billed for whichever model was actually used. LiteLLM has an equivalent router fallback.

The discipline: keep the two axes separate. Cost routing is "I proactively pick the cheap one"; fallback is "passively keep me from going down." Mistaking a degraded fallback for a cost optimization will quietly drop your quality during a provider hiccup without you noticing.

Hands-on

# Cost routing: classify in one shot, then pick the tier
tier = classify(task)                       # judge difficulty with a cheap model
model = "smart" if tier == "hard" else "cheap"

# Availability fallback: OpenRouter models array, ordered backup
client.chat.completions.create(
    model="anthropic/claude-opus-4-8",
    extra_body={"models": [          # primary errors → fall through
        "anthropic/claude-sonnet-4-6",
        "openai/gpt-latest",
    ]},
    messages=[...],
)
# The response's model field tells you which was actually used → always log it

Two Cache Layers: Provider Prompt Cache + Semantic Cache

Background & Principle

Layer 1 · provider prompt caching: it caches the prefix. Mark the stable, unchanging parts (system prompt, tool definitions, long documents) with cache_control, and the provider caches that KV computation; subsequent hits are charged at a tiny read rate. Anthropic's docs are explicit: cache read is ~0.1× the base input price, a 5-minute cache write ~1.25×, and a 1-hour write ~2×. This layer is a money-saver when you repeatedly call with the same long prefix (agent loops, multi-turn conversations especially), but it only reuses computation—it doesn't skip inference; the model is still genuinely called each time.

Layer 2 · semantic cache (GPTCache and similar): it caches the whole Q&A pair. A new query is first embedded and compared by similarity against past queries; if similar enough, the old answer is returned directly and the model is not called at all—both latency and cost drop to ~0. The cost is that the "similar" judgment is risky: a loose threshold returns answers it shouldn't.

The two layers are orthogonal and stackable: the prefix cache lowers the cost per call; the semantic cache lowers the number of calls. The former is near-zero-risk; the latter saves more but you must manage the threshold.

Hands-on

# Layer 1: Anthropic prompt caching — tag the stable prefix
client.messages.create(
  model="claude-opus-4-8",
  system=[{
    "type": "text",
    "text": LONG_STABLE_INSTRUCTIONS,          # long and identical each time
    "cache_control": {"type": "ephemeral"}  # ← cached, read ≈ 0.1×
  }],
  messages=[{"role":"user","content": today_query}],  # variable part goes last
)

# Layer 2: semantic cache — a hit skips the model (GPTCache idea)
emb = embed(query)
hit = vec_store.search(emb, threshold=0.95)   # threshold is the key knob
if hit: return hit.cached_answer            # 0 tokens, 0 latency
ans = call_llm(query); vec_store.add(emb, ans)

One rule: put volatile content at the end of the prefix. A prompt-cache hit requires the prefix to be byte-for-byte identical—a dynamic timestamp at the top of the system prompt can invalidate the entire cached block.

Observability & Key Governance: See What You Spend, Control Who Spends

Background & Principle

Observability: LLM-call observability differs from ordinary API monitoring—you want to see prompt/completion pairs, token usage, per-call cost, latency, and (for agents) the full call chain. Langfuse is the individual's go-to open-source option: self-hostable, natively understands LLM concepts (tokens, cost, traces, nested spans), and integrates with OpenTelemetry and the major SDKs. Once it's in, "which side project quietly burned $40 this month" goes from guesswork to a chart.

Key governance—three iron rules: (1) real keys never enter code or git, only env vars or a secret manager, and only at the gateway layer. (2) issue each app a virtual key instead of sharing a real one: LiteLLM's virtual keys support per-key budgets, rate limits, model allowlists, and expiry—an app that runs away only burns its own quota and can be revoked alone without affecting others. (3) start budgets small ($5–50), raise by real usage, and set a hard cap on every key—the individual's only backstop against "a 3 a.m. infinite loop blows up the bill."

Hands-on

# Langfuse: decorator auto-traces, zero intrusion
from langfuse.openai import OpenAI   # drop-in replacement, auto-records token/cost/latency
client = OpenAI(base_url="http://localhost:4000/v1", api_key="sk-my-virtual-key")

# LiteLLM virtual key: per-app budget + model allowlist + expiry
# POST /key/generate
{
  "key_alias": "side-project-blog",
  "max_budget": 20,            # this key spends at most $20 ever
  "budget_duration": "30d",     # reset every 30 days
  "models": ["cheap"],          # cheap tier only, prevents misuse of opus
  "duration": "90d"             # the key itself expires in 90 days
}