Local & Edge LLM

Quantization Is an Engineering Decision, Not a Free Lunch

Background & Principle

First-principles memory formula: weight bytes = param count × bytes per param. FP16 is 2 bytes/param, so a 7B model is 14GB raw; quantize to 4-bit (~0.5 byte/param incl. scale overhead) and it drops to ~4GB — that's the precondition for running locally at all. GGUF's K-quants (Q4_K_M, Q5_K_M) are today's de-facto standard: they use different precision for different tensors (higher bits for critical attention and feed_forward layers), and Q4_K_M is the recognized size/quality sweet spot. Layer on imatrix (importance matrix): a calibration corpus measures "which weights, when nudged, move the output most," and quantization protects those first — driving perplexity loss lower still.

But the real trap is the metric. The community habitually measures quantization loss with perplexity, and PPL is the average log-likelihood of next-token — it is deeply insensitive to "occasionally picking one critical token wrong." Yet exactly that occasional error is what fails a compile or snaps a reasoning chain. So you'll see "Q4's PPL only rose 1%" while real coding / math tasks drop a whole tier. The sensitivity of multi-step reasoning and precise generation to quantization is systematically understated by an averaged PPL.

Hands-on

# Don't trust PPL alone — A/B with samples from YOUR task
# Pull two quant levels of the same model, run your real 20 evals
ollama pull qwen3:8b              # default is usually Q4_K_M
ollama pull qwen3:8b-q8_0         # high-precision control

# Quantize with your own imatrix (max fidelity):
# 1) compute importance matrix on domain-representative text
./llama-imatrix -m model-f16.gguf -f calib.txt -o imat.dat
# 2) attach --imatrix at quantize time; critical weights are protected
./llama-quantize --imatrix imat.dat model-f16.gguf model-Q4_K_M.gguf Q4_K_M

The discipline: pick the quant level by task, not by "it runs." Forgiving tasks like extraction / classification tolerate Q4 or lower; code / math / long-chain reasoning warrant Q5_K_M / Q8, or a smaller-but-full-precision model.

Memory Math, and Judging "Good Enough Locally"

Background & Principle

"Will this model run on my machine" isn't a lookup, it's arithmetic. The budget has three parts:

• Weights: param count × bytes per param (see §1). A 70B at Q4 ≈ 40GB.
• KV cache: grows linearly with context length, roughly 2 × layers × kv_heads × head_dim × seq_len × bytes. This is the underestimated chunk — a 70B at 32K context can eat another 10GB+ in FP16 KV cache. GQA (grouped-query attention) shrinks kv_heads, which is why modern models can hold long context at all.
• Overhead: activations, compute graph, the framework itself — leave 1–2GB headroom.

Sum these three against your VRAM (or Apple Silicon unified memory) to know "does it fit." More important is the second judgment: fits ≠ should use. Local models have a clear "good-enough zone" and a "don't-touch zone":

Hands-on

# 30-second estimate: does the model fit?
weights_GB = params_B * bytes_per_param   # Q4≈0.5, Q8≈1.0, FP16≈2.0
kv_GB      = 2 * layers * kv_heads * head_dim * ctx * 2 / 1e9  # FP16 KV
need_GB    = weights_GB + kv_GB + 1.5     # +overhead headroom

# e.g. Llama-70B Q4, 32K context, GQA(kv_heads=8)
#  weights ≈ 40GB, kv ≈ 10GB+ → need ≈ 52GB
#  → a 48GB machine "that runs 70B" can't actually open 32K context

Runtimes Aren't Equal: llama.cpp / Ollama / MLX / LM Studio

Background & Principle

"Running a model locally" is really a layered stack, each layer solving something different:

• llama.cpp: the core C++ inference engine, origin of the GGUF format, cross-platform (CPU/CUDA/Metal). Constrained decoding (GBNF) and imatrix quantization live here. For fine control, use it directly.
• Ollama: adds a model registry, Modelfile config, and an OpenAI-compatible local server (localhost:11434) on top of llama.cpp — its biggest value, letting you flip cloud code to local by changing one base_url.
• MLX / mlx-lm: Apple's framework for its own silicon, fully exploiting the unified-memory architecture (CPU/GPU share one physical address space, zero copies). For memory-bandwidth-bound inference on a Mac, throughput typically beats routes that adapt through a Metal layer. Worth a dedicated try for Mac users.
• LM Studio: a GUI shell with a llama.cpp or MLX backend — good for the exploration phase if you'd rather avoid the command line.

Hands-on

Ollama's most insidious trap: a small default num_ctx (historically often 2048/4096). You think you're using the model's 128K window, but it only sees a few thousand tokens and silently drops the rest — "unexplained amnesia" on long-context tasks is this nine times out of ten. Raise it explicitly via the Modelfile:

# Modelfile: set context explicitly, don't eat the silent default
FROM qwen3:8b
PARAMETER num_ctx 32768

# create and serve
ollama create qwen3-32k -f Modelfile

# then point the OpenAI SDK straight at local — almost no code change
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
r = client.chat.completions.create(model="qwen3-32k",
        messages=[{"role":"user","content":"..."}])

The Boundary of Small-Model Agents, and Hybrid Routing

Background & Principle

Drop an 8B local model into an agentic loop and the usual collapse is unstable tool-call format: a missing bracket, truncated JSON, a misspelled field — one parse error kills the whole loop. Cloud models do this so well you forget how hard it is; small models don't have that luxury. The fix isn't "ask the model to behave," it's constrained decoding: llama.cpp's GBNF grammar / JSON Schema zeroes out the probability of any grammar-violating token at the sampling layer, physically guaranteeing valid JSON. That's an order of magnitude more reliable than writing "please output JSON" in the prompt.

But mind the boundary: it guarantees syntax, not semantics. The model can still fill in a syntactically valid but factually wrong value. And in llama.cpp's implementation, the schema only constrains output — it is not injected into the prompt; the model can't "see" the schema, so if you want it to understand field meanings, you still describe them in the prompt.

The real engineering answer is usually hybrid: the local model does the high-volume, forgiving, privacy-sensitive grunt work (routing classification, PII redaction, first-pass filtering, drafting), and the hard, low-frequency, must-be-best steps go to a cloud frontier model. Local serves as router and fallback, not the full-time lead.

Hands-on

# Constrain local output with JSON Schema (llama.cpp server / llama-cpp-python)
# the sampling layer forces valid JSON — even small models can't emit garbage
schema = {"type":"object",
  "properties":{"intent":{"enum":["search","refund","other"]},
                "urgent":{"type":"boolean"}},
  "required":["intent","urgent"]}

r = client.chat.completions.create(model="qwen3-8k",
      messages=[{"role":"user","content": ticket}],
      extra_body={"response_format":{"type":"json_schema",
                  "json_schema":{"schema": schema}}})
# intent=search → handle locally; intent=refund & urgent → escalate to cloud