AI/ML Deep Dive: Agent Architectures

Think of it as a script wrapped in a while loop — on each iteration the LLM first "thinks about what to do" (Thought), then "calls a tool" (Action), and once it sees the tool's output (Observation) it decides the next step, repeating until it can produce a final answer.

A bare LLM is strictly "one input, one output" — it answers whatever you ask but can't reach external tools or adjust strategy based on intermediate results. Plenty of real tasks ("look up Apple's stock price today and compute how much it has moved since the start of the year") require chaining: call an API, do some math, maybe fetch more data. ReAct interleaves reasoning with acting, turning the LLM into something like a programmer who can see intermediate results and react to them. It's the ancestor of nearly every modern agent framework (LangChain Agent, LangGraph, OpenAI function calling).

The core is a Thought → Action → Observation loop, all driven through prompting so the LLM itself emits both Thought and Action:

On each round the LLM sees the full history (all prior Thought / Action / Observation entries are concatenated back into the prompt) and decides the next move. Stopping conditions: the LLM emits "Final Answer: …" itself, or a step cap is hit.

from langchain.agents import create_react_agent, AgentExecutor
from langchain.tools import tool
from langchain_openai import ChatOpenAI

# 1) Define tools — no different from writing ordinary Python functions
@tool
def get_stock_price(symbol: str) -> float:
    """Look up the current price of a stock"""
    return 188.42  # simplified: in real life call yfinance/Bloomberg

@tool
def calculator(expression: str) -> float:
    """Evaluate a math expression"""
    return eval(expression)

# 2) Assemble the agent — LangChain auto-injects the ReAct prompt template
agent = create_react_agent(
    llm=ChatOpenAI(model="gpt-4o"),
    tools=[get_stock_price, calculator],
    prompt=hub.pull("hwchase17/react")  # the classic ReAct prompt
)
executor = AgentExecutor(agent=agent, tools=[get_stock_price, calculator])

# 3) Run — you only supply the goal; the framework owns the loop
result = executor.invoke({"input": "How much is Apple up today? It opened the year at 165."})

• Original paper: ReAct: Synergizing Reasoning and Acting — includes a visualized demo
• Lilian Weng's LLM Powered Autonomous Agents — the agent landscape

Think project manager + grunt worker — one LLM call "plans the full list of steps" (like writing a TODO list), then another LLM call "executes the list one item at a time". Splitting "think it through" from "do it" prevents the ReAct trap of forgetting the big picture after two steps.

ReAct's core weakness is myopia — it only thinks one step at a time, so on a 10+ step task ("plan next week's department quarterly review for me") it tends to wander, do useless work, and call the same tool repeatedly. Plan-and-Execute forces the LLM to first decompose the global task into an ordered set of subtasks, then execute one by one, updating the plan as it goes. This "plan first, then act" paradigm comes from classical AI planning (STRIPS, PDDL); combined with LLMs it becomes general-purpose without hand-coded domain knowledge.

The key contrast with ReAct: ReAct makes "independent decisions per step", while Plan-and-Execute has an explicit plan state driving the system. That plan can be persisted, audited, or edited by a human — which matters a lot in production.

from langgraph.prebuilt import create_react_agent
from langgraph.graph import StateGraph
from langchain_openai import ChatOpenAI

planner = ChatOpenAI(model="gpt-4o")
executor_agent = create_react_agent(llm, tools=[search, calc, email])

def plan(state):
    # Planner emits the full ordered step list in one shot
    plan = planner.invoke(f"Break this goal into ordered steps: {state['goal']}")
    return {"plan": plan.steps, "done": []}

def execute(state):
    step = state["plan"][0]
    result = executor_agent.invoke({"input": step})
    return {"plan": state["plan"][1:],
            "done": state["done"] + [(step, result)]}

def replan(state):
    # Look at history and decide whether to amend remaining plan
    return planner.invoke(f"Done: {state['done']}, remaining: {state['plan']}, change anything?")

graph = StateGraph().add_node(plan).add_node(execute).add_node(replan)
graph.add_conditional_edges("execute", lambda s: "END" if not s["plan"] else "replan")

• LangGraph's Plan-and-Execute tutorial — engineering deep dive
• Paper: Plan-and-Solve Prompting — the predecessor paradigm

Like coding against unit tests — when one fails you don't immediately switch strategies; first you jot down a "reflection note" (why did I get this wrong?), then feed the note into your next attempt's prompt. Reflexion = fail → self-critique → learn → retry.

ReAct and Plan-and-Execute share a flaw: once they fail, they fail — they don't learn from the mistake. If the first attempt is wrong, the second one usually walks down the same wrong path. Reflexion's insight: explicitly write "why did I fail?" into the next prompt as extra context, and model performance improves significantly — much like a student keeping an "error notebook" so they dodge the same trap next time. This "self-reflection as short-term memory" is a cheap way to boost agent performance without retraining.

The three pieces: Actor (does the task), Evaluator (judges success), Reflector (writes the reflection). Reflections don't update model weights; they live only in the prompt — so this is "short-term memory", more like giving a colleague review feedback for them to redo their work, not re-training them.

def reflexion_loop(task, max_trials=3):
    memory = []  # reflection notes (short-term memory)

    for trial in range(max_trials):
        # 1) Actor attempts the task with reflection notes
        prompt = f"""Task: {task}
Past reflection notes (avoid repeating the same mistakes):
{chr(10).join(memory)}"""
        output = actor_llm.invoke(prompt)

        # 2) Evaluator decides pass/fail (unit tests for code; LLM judge otherwise)
        success, error_msg = evaluator(output)
        if success:
            return output

        # 3) Reflector writes a reflection note
        reflection = reflector_llm.invoke(f"""
You just attempted: {task}
Your output: {output}
Eval result: {error_msg}
Briefly reflect: why did it fail? what should you change next time?
""")
        memory.append(reflection)

    return output  # hit the cap without success

• Original paper: Reflexion: Language Agents with Verbal Reinforcement Learning
• LangGraph's Reflection / Reflexion tutorial

Imagine packaging ReAct + a filesystem + a memory store + a subtask queue into "a main function that loops forever" — you toss a one-liner goal in and the agent decomposes tasks, calls tools, records intermediate results, and keeps going until it decides the goal is done (or it maxes out your credit card). AutoGPT was the 2023 open-source project that ignited the "autonomous agent" concept and set the design template that every autonomous agent since follows.

Earlier agents needed humans in the loop at every step, or could only handle short tasks. AutoGPT tried to answer: "Can you give the LLM a long-horizon goal (like 'gather every AI startup funding announcement this month and write it up') and let it run fully autonomously?" To do it, AutoGPT stitches several capabilities together: (a) task queue management (todo / done), (b) file read/write (a long-term workspace), (c) internet access, (d) a vector memory store (remembering key info across steps), (e) subgoal decomposition. It invented no new algorithms; its value is the engineering paradigm — "wrap the LLM as a process with persistent state".

Note its relationship to the prior three architectures: each inner step uses ReAct for decisions; the task queue gives Plan-and-Execute-style orchestration across steps; you can layer Reflexion on failures. So AutoGPT isn't a replacement — it's the integrative engineering template: a reference implementation for "continuously running autonomous agents".

# Minimal AutoGPT main loop — real AutoGPT is the productionized version
def autogpt(goal, max_steps=50, budget_usd=5):
    task_queue = [goal]
    memory = VectorMemory()  # Chroma/FAISS instance
    workspace = FileSystem("./workspace")
    cost = 0.0

    for step in range(max_steps):
        if not task_queue or cost > budget_usd:
            break

        task = task_queue.pop(0)
        context = memory.search(task, k=5)  # recall relevant history

        # LLM decides the action for this step (structured output)
        decision = llm.invoke(prompt=f"""
You are {agent_persona}, goal: {goal}
Current subtask: {task}
Relevant memory: {context}
Output JSON: {{"action": "search|write_file|spawn_task|done",
              "args": ...}}
""")

        # Execute + write to memory / task queue / filesystem
        result = execute_action(decision, workspace)
        memory.add(f"step{step}: {task} → {result}")
        cost += estimate_cost(decision)

        if decision["action"] == "spawn_task":
            task_queue.extend(decision["args"]["subtasks"])
        elif decision["action"] == "done":
            break

• AutoGPT GitHub project — read the original README for design philosophy
• Anthropic's Building Effective Agents — reflections on the boundaries of autonomous agents