Memory & State Management

Split State Into Four Layers — Mixing Them Is the Accident

Background & Principles

Chapter 1 of any OS textbook tells you memory is tiered: registers / cache / RAM / disk. An LLM's "memory" is also tiered — most people just don't realize it. Anthropic's Building Effective Agents repeats one line: "context is a budget" — you must actively decide what each token is spent on. The four classes below should be tracked separately:

• Conversation State: the message history of the current session. Lifecycle = one session. High trust (the user just said it). Write is append-only, read is usually reverse-chronological or full.
• Scratchpad / Working Memory: intermediate output of an in-progress task — tool calls, reasoning steps, partial plans. Lifecycle = one task. Burn after reading; almost no scratchpad should survive across tasks.
• User Profile: stable facts about the user across sessions — name, preferences, long-term goals, prior decisions. Lifecycle = the user's lifetime. Trust needs validation (what the user said may be stale). Write requires LLM extraction + dedup + conflict resolution; read usually means full-inject into system prompt.
• Knowledge Base: domain knowledge unrelated to a specific user — docs, codebase, product spec. Lifecycle = the knowledge's own version. Read must be query-aware retrieval (that's where RAG truly shines).

Cramming all four into one mechanism (stuff a vector DB) hits three failures simultaneously: (1) embedding-based retrieval over conversation loses temporal ordering — "the previous message" may not even rank; (2) user profile drowns in 1M chunks, cosine similarity can't pick the key fact; (3) scratchpad persists forever and pollutes the next task. The right answer is independent storage + independent read/write policy per layer.

Hands-on Example

Model the four layers explicitly as Python classes; force prompt assembly through the layered interface:

# memory_layers.py — force the split; ban "one dict for everything"
from dataclasses import dataclass, field
from typing import Protocol

class MemoryLayer(Protocol):
    def read(self, query: str) -> str: ...
    def write(self, content: str, meta: dict) -> None: ...

@dataclass
class Conversation:                # short-term, append-only
    messages: list = field(default_factory=list)
    def read(self, _): return self.messages[-20:]   # sliding window default

@dataclass
class Scratchpad:                  # per-task scope
    by_task: dict = field(default_factory=dict)
    def clear(self, task_id): self.by_task.pop(task_id, None)

@dataclass
class Profile:                     # cross-session stable facts
    facts: dict = field(default_factory=dict)  # {"name":"BigCat", ...}
    def read(self, _): return self.facts            # full

@dataclass
class Knowledge:                   # docs/code, RAG entry
    index: object   # your vector store
    def read(self, query): return self.index.search(query, k=5)

# —— assemble through the four layers ——
def build_context(query, conv, scratch, profile, kb):
    return {
        "system": f"User profile:\n{profile.read(None)}",
        "messages": conv.read(None),
        "working": scratch.by_task.get(current_task_id, []),
        "retrieved": kb.read(query),         # only what matches query
    }

The point isn't the code — it's that the code forces you to answer four questions: which profile fields go fully into system? when does scratch clear? how large is the conv window? what's k for kb retrieval? An agent whose author can't answer all four will ship a memory incident.

Short-Term Memory in Three Tiers: When to Truncate / Summarize / Go Hierarchical

Background & Principles

Context overflowed — what now? Three tiers, different complexity and loss:

• Truncate (sliding window): keep the last N messages, drop the rest. Zero cost, zero loss, reproducible. Use for: support, Q&A, single-turn instruction — conversations where old messages have no future value.
• Summarize (incremental compaction): at some threshold, compress old messages into a summary that sits at the top of the system prompt. Medium loss, medium cost. Use for: research, planning, creative writing — anywhere you may need to look back. Claude Code and ChatGPT long chats both do this (Anthropic calls it "compaction"; we covered it in Day 2).
• Hierarchical (chunk + retrieve): old messages get chunked into a vector store; query-aware retrieval pulls them back when needed. Low loss, but architecturally complex. The MemGPT paper (Packer et al. 2023) opened this lane: "main context" + "external context" + self-edit functions let the LLM page in/out itself.

The selection rule is simple — "will I ever need to look up specific old details?" Support: usually no (truncate). Long-form writing: yes, but inexact (summarize). Investment research / code review: yes, with exact quotes (hierarchical).

Hands-on Example

All three tiers can live in one manager that auto-upgrades by message count:

# conversation_manager.py — three tiers, auto switch
class ConversationManager:
    def __init__(self, mode="truncate", window=20, summary_threshold=40):
        self.messages, self.summary = [], None
        self.mode, self.window, self.threshold = mode, window, summary_threshold

    def add(self, msg): self.messages.append(msg)

    def build(self):
        if self.mode == "truncate":
            return self.messages[-self.window:]

        if self.mode == "summarize":
            if len(self.messages) > self.threshold:
                old = self.messages[:-self.window]
                self.summary = compact(self.summary, old)  # LLM incremental
                self.messages = self.messages[-self.window:]
            head = [{"role":"system","content":f"Prior summary:\n{self.summary}"}] \
                   if self.summary else []
            return head + self.messages

        if self.mode == "hierarchical":
            recent = self.messages[-self.window:]
            old_chunks = self.vector.search(latest_user_msg(recent), k=5)
            return [recall_block(old_chunks)] + recent

# —— compaction prompt (don't lose the important details) ——
COMPACT = """Summarize the prior conversation. PRESERVE:
- All factual decisions made
- All open questions / TODOs
- Any explicit user preferences mentioned
DISCARD: greetings, clarifications already resolved, model's apologies.
Existing summary: {old_summary}
New messages: {new_msgs}
Output (≤300 tokens):"""

The PRESERVE / DISCARD list in the compaction prompt is what determines summary quality. Anthropic's docs explicitly call out "preserve decisions and open TODOs" — exactly the two things research and coding sessions lose most often.

Three Long-Term Memory Architectures: Vector / Structured KV / Episodic Event Log

Background & Principles

Treating a vector DB as universal long-term memory was a 2023 myth. The senior pattern is splitting by query mode into three stores, each doing its own job:

• Vector store (semantic retrieval): great for "find similar" / "find relevant" open-ended queries. Use for: docs, fuzzy chat recall, case-based reasoning. Fails at: exact facts, list enumeration, temporal queries.
• Structured KV / SQL (exact facts): great for "what is the value of X" or "list all Y". Use for: user preferences, entity attributes, structured profile. Fails at: fuzzy queries. MemGPT's core memory is essentially structured KV.
• Episodic event log (temporal events): great for "what happened last time", "what did I do in the past two weeks", "why did the last attempt fail". An append-only time series, indexed by (user_id, ts). Use for: reflection, postmortem, decision trace. The reflective memory in Reflexion (Shinn et al. 2023) is this structure.

These three are not mutually exclusive — they are complementary. A mature agent has all three simultaneously: vector for docs + KV for profile + event log for behavior history. Letta (the productized MemGPT) calls this the "memory hierarchy" and explicitly distinguishes core_memory (KV), archival_memory (vector), and recall_memory (event log). Microsoft GraphRAG takes a fourth path (knowledge graph), suitable for entity-relation-dense domains, but with high overhead — most teams should nail the first three first.

Hands-on Example

Implement a personal agent that "remembers you" with all three stores; the key is write routing — given an incoming utterance, decide which store it goes to:

# memory_router.py — one utterance in, routed to the right store
ROUTE_PROMPT = """Classify this user utterance into ONE memory type:
- FACT      : a stable fact about the user (name, role, preference)
- EVENT     : an action/decision/experience that happened
- DOCUMENT  : reference content (article, code, doc)
- NONE      : transient (greeting, ack, clarification)
Return JSON: {"type": "...", "extract": "..."}"""

def ingest(utterance, user_id):
    r = llm(ROUTE_PROMPT, utterance)
    if r["type"] == "FACT":
        profile_kv.upsert(user_id, r["extract"])      # structured KV
    elif r["type"] == "EVENT":
        event_log.append(user_id, ts=now(), text=r["extract"])
    elif r["type"] == "DOCUMENT":
        vector_store.add(embed(r["extract"]), meta={"user":user_id})
    # NONE → drop

# —— on read, route by query type ——
def recall(query, user_id):
    intent = classify(query)  # fact / temporal / semantic
    if intent == "fact":
        return profile_kv.get_all(user_id)            # full profile
    if intent == "temporal":
        return event_log.range(user_id, last="7d")    # by time
    return vector_store.search(query, k=5, filter={"user":user_id})

The essence of this router — the question chooses the store. "What's my name?" → KV. "What did we talk about last week?" → event log. "Find a passage I wrote about meditation" → vector. One store for everything is a beginner's illusion.

Self-Maintained User Profile: Let the LLM Edit Its Own Notes "About You"

Background & Principles

Section 3's structured KV solves "what to store and how to read"; one engineering question remains — who decides what to write. Three options:

• User explicitly declares: have the user fill a form or write a system prompt. Low coverage, annoying.
• Rule extraction: regex / rules to spot "my name is X" / "I like Y". Brittle, low recall.
• LLM self-maintained: give the LLM an update_profile tool; after each turn it decides whether to update. MemGPT / Letta / ChatGPT memory / Claude memory all take this path.

The hard part of self-maintained isn't "let it write" — it's "let it write the right thing". The profile is fully injected into the next session's system prompt; garbage in it poisons forever. Four problems to solve:

1. Granularity: "user mentioned they went jogging" — write or not? Too fine → profile explodes; too coarse → miss the key. Heuristic: only write facts that remain true across sessions.
2. Conflict resolution: when new contradicts old ("I'm vegetarian" → "I eat meat now"), overwrite or append? Default: overwrite + keep a changelog.
3. Forgetting: profile cannot grow unbounded. LRU / explicit TTL / user-initiated delete. Letta uses "archival" to move inactive facts into the vector store.
4. Hallucination filter: the LLM may infer facts the user never stated ("user mentioned their kid's homework → assume they have a school-aged child"). Every self-maintained write must cite a specific message ID as evidence; no evidence, no write.

Hands-on Example

Give the agent a profile tool; after each turn run a "does this need updating" check:

# profile_tools.py — self-maintained profile tool for the LLM
PROFILE_UPDATE_PROMPT = """Review the latest user message and decide if the user
profile should be updated. Only write facts that:
1. Are explicitly stated by the user (cite message)
2. Are likely to be true beyond this session
3. Are not already in the profile (or contradict it)

Current profile: {profile_json}
Latest message:  {message}

Output JSON:
{
  "action": "add" | "update" | "delete" | "none",
  "key": "...",
  "value": "...",
  "evidence_msg_id": "...",   // required; no evidence → "none"
  "reason": "..."
}"""

def maybe_update_profile(user_id, latest_msg, msg_id):
    current = profile_kv.get_all(user_id)
    decision = llm(PROFILE_UPDATE_PROMPT.format(
        profile_json=json.dumps(current), message=latest_msg))
    if decision["action"] == "none": return
    if not decision.get("evidence_msg_id"): return  # no evidence → reject
    profile_kv.apply(user_id, decision)
    changelog.append(user_id, decision, ts=now())   # traceable

# —— inject profile at session start ——
def build_system_prompt(user_id):
    p = profile_kv.get_all(user_id)
    return f"""You are a personal assistant for the following user.
Stable facts about them (use to personalize, but verify before acting):
{json.dumps(p, indent=2, ensure_ascii=False)}

Important: if a fact seems outdated, ASK the user to confirm rather than
silently override the profile."""

Three details decide quality: mandatory evidence_msg_id (blocks hallucination), changelog (traceable — when the user asks "how do you know I'm vegetarian" you can answer), and a system prompt that pushes the model to verify (so it doesn't act overconfidently on stale profile). All three together turn the profile from toy into engineered artifact.