Chapter 5: Memory — Giving the Agent a Past

Ravi's review-sorting agent from Chapter 2 is working great. His boss is impressed. Then she asks for a new feature: "Can it handle a conversation? Like, I paste five reviews, and then I ask 'which one was about the battery?'" Ravi says, "Sure, easy." He runs the agent on each review, stores the results, and... waits. The agent sorted each review fine. But when the boss asks "which one was about the battery?", the agent has no idea what she's talking about. It doesn't remember the five reviews. Each call was a fresh start. "It's the goldfish thing," Ravi mutters, remembering Chapter 2. "The brain is stateless." "So give it a memory," Priya says, leaning over again. "You already know how. You just haven't wired it up yet."

Why This Matters

By the end of this chapter, you'll have built two kinds of memory for your agent: short-term (the conversation so far, replayed each call) and long-term (facts stored outside the conversation, retrieved when needed). You'll also know what to do when the conversation gets too long for the context window — the problem that every production agent eventually faces.

Recap: The Goldfish Problem

From Chapter 2: the LLM is stateless. Every call to client.chat.completions.create() starts fresh. The model forgets everything from the last call. The only reason ChatGPT seems to remember your conversation is that the application sends the whole history back with every new message.

Short-term memory isn't magic. It's just the messages list, growing each turn.

This is short-term memory: the conversation so far, stored in your code as the messages list, replayed to the LLM on every call. Our agent loop from Chapter 3 already does this! The messages list grows with each turn, and the LLM sees the whole thing each time.

# Our agent loop already has short-term memory — the messages list
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Hi, I'm Ravi."},
    {"role": "assistant", "content": "Hi Ravi! Nice to meet you."},
    {"role": "user", "content": "What's my name?"},
]
# The LLM sees all four messages and answers "Ravi" — because the history is right there.

// Short-term memory = the messages list. It grows each turn. The LLM reads it all every call.

The Problem: The Context Window Fills Up

Short-term memory works great — until it doesn't. Remember the context window from Chapter 2? Every model has a maximum number of tokens it can hold in one call. As the conversation grows, the messages list grows, and eventually you hit the limit.

The context window is finite. Long conversations will outgrow it.

For gpt-4o-mini, the context window is 128,000 tokens — that's a lot, but a long agent session with tool results can blow through it. For older or smaller models, the limit might be 8,000 or 16,000. Either way, the day comes when the conversation doesn't fit.

What do you do? Three strategies, from simplest to most sophisticated:

Strategy 1: Truncate — drop old messages

The simplest fix: when the messages list gets too long, drop the oldest messages. Keep the system message and the most recent turns; throw away the middle.

def trim_messages(messages, max_messages=20):
    # Always keep the system message (first) and recent messages
    system = messages[0]
    rest = messages[1:]

    if len(rest) > max_messages:
        rest = rest[-max_messages:]  # keep only the most recent
return [system] + rest

// Simple but brutal: old context is gone. The agent forgets early turns.

Watch it! Truncation is simple but dangerous. If the agent needs a fact from turn 3 to answer a question in turn 30, and you've dropped turn 3, the agent will hallucinate or say "I don't know." Truncation works for chitchat; it fails when early context matters.

Strategy 2: Summarise — compress old messages

Instead of dropping old messages, summarise them. Use the LLM itself to compress the first part of the conversation into a short summary, then keep the summary plus the recent turns.

Old turns become a summary. Recent turns stay verbatim. The agent keeps the gist without the tokens.

def summarise_old_messages(messages, keep_recent=6):
    system = messages[0]
    old = messages[1:-keep_recent]
    recent = messages[-keep_recent:]

    if not old:
        return messages  # nothing to summarise
# Ask the LLM to summarise the old conversation
    summary_response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Summarise this conversation, keeping key facts and decisions."},
            {"role": "user", "content": str(old)},
        ]
    )
    summary = summary_response.choices[0].message.content

    return [
        system,
        {"role": "system", "content": f"Summary of earlier conversation: {summary}"},
    ] + recent

// The old turns are replaced by one summary message. Recent turns stay verbatim.

Strategy 3: External store — long-term memory

For facts that need to survive beyond the conversation — the user's name, their preferences, things they told the agent last week — you don't keep them in the messages list at all. You store them outside, in a database or file, and retrieve them when needed.

Long-term memory is a store the agent reads from and writes to. It's not in the messages list.

Here's a simple long-term memory store using a JSON file. The agent can save facts about the user and retrieve them later — even in a completely new conversation.

import json
from pathlib import Path

class MemoryStore:
    def __init__(self, path="memory.json"):
        self.path = Path(path)
        if self.path.exists():
            self.data = json.loads(self.path.read_text())
        else:
            self.data = {}

    def save(self, key, value):
        self.data[key] = value
        self.path.write_text(json.dumps(self.data, indent=2))

    def get(self, key, default=None):
        return self.data.get(key, default)

    def all_facts(self):
        return self.data

// A simple key-value store. In production this might be a database, Redis, or a vector store.

Now the agent can use tools to save and retrieve facts. We give it two memory tools:

def remember_fact(key: str, value: str) -> str:
    """Save a fact about the user for later."""
    memory.save(key, value)
    return f"Saved: {key} = {value}"
def recall_facts() -> str:
    """Retrieve all known facts about the user."""
    facts = memory.all_facts()
    if not facts:
        return "No facts stored yet."
return "\n".join(f"- {k}: {v}" for k, v in facts.items())

// Two tools: one to write to long-term memory, one to read from it. The LLM calls them when it needs to.

Now watch the agent in action across two separate sessions:

# Session 1
User: "Hi, I'm Ravi. I prefer concise answers."
Agent THINK: "I should remember this."
Agent ACT:   remember_fact(key="name", value="Ravi")
Agent ACT:   remember_fact(key="preference", value="concise answers")
Agent: "Got it, Ravi."
# Session 2 — completely new conversation, messages list is empty
User: "What do you know about me?"
Agent THINK: "I should check my memory."
Agent ACT:   recall_facts()
Agent OBSERVE: "- name: Ravi\n- preference: concise answers"
Agent: "You're Ravi, and you prefer concise answers."

// The agent remembered across sessions — because the facts live in the store, not in the messages.

Note Long-term memory is just tools that read and write a store. The LLM decides when to save a fact ("the user told me their name — I should remember it") and when to retrieve ("the user asked what I know — I should check memory"). The memory isn't magic; it's two more tools in the loop.

There Are No Dumb Questions

Q: How does the LLM know WHEN to save a fact? Do I have to tell it?

A: The tool description does the work. If remember_fact says "Save a fact about the user when they share personal info, preferences, or instructions for later," the LLM will call it when the user says "I'm Ravi" or "I prefer concise answers." You steer this with the description, just like any tool.

Q: What if the agent saves wrong facts, or too many?

A: This is a real problem in production. Agents can over-eagerly save trivia, or save contradictory facts. Solutions include: limiting what the agent can save (structured keys), deduplication, and giving the agent a "forget" tool. We'll revisit this in Chapter 10 (Guardrails) — memory management is a safety concern, not just a feature.

Q: Is this the same as RAG from Chapter 7?

A: Related but different. This is simple key-value memory — exact lookup by key. RAG (Chapter 7) is for when you have thousands of documents and need semantic search — finding the right document by meaning, not by key. Long-term memory and RAG are two ends of a spectrum; real systems blend both.

Where People Come Unstuck

Mistake #1: Treating short-term memory as unlimited

The messages list grows. Every turn, every tool result, every system message adds tokens. A long agent session with large tool results can blow through the context window faster than you think. Always have a plan for when the conversation gets long — truncate, summarise, or move to external storage.

Mistake #2: Forgetting that tool results count toward the limit

Tool results go into the messages list too. If your search_books tool returns a 5,000-token catalogue listing, that's 5,000 tokens now sitting in your context window. Tools that return huge results eat your context budget fast. Keep tool results concise — return summaries, not dumps.

Mistake #3: Storing everything in the conversation

If the user tells the agent their name on turn 1, and you need that fact on turn 50, it has to survive 48 turns of context. If you truncate, it's gone. Move durable facts to long-term storage early. The conversation is for the current task; the store is for permanent facts.

Brain Power

Think about a customer support agent for an online store. It handles conversations like "where's my order?" and "I want to return this."

What facts should go in short-term memory (the conversation)? What facts should go in long-term memory (the store)? What might you summarise? And what tool results might be large enough to eat your context budget — and how would you handle that?

There's no single right answer. The point is to start thinking in memory tiers — what lives where, and why.

Chapter Summary

Short-term memory is the conversation so far — the messages list, replayed to the LLM on every call. Our agent loop already does this.
The context window is finite. Long conversations, large tool results, and chatty agents will eventually outgrow it.
Three strategies for a full context: truncate (drop old messages — simple but lossy), summarise (compress old turns into a summary — keeps the gist), or external store (move durable facts out of the conversation entirely).
Long-term memory is a store outside the conversation — a file, database, or vector store. The agent reads and writes it via tools.
Memory tools — remember_fact and recall_facts — let the agent save and retrieve facts across sessions. The LLM decides when to use them based on the tool descriptions.
Keep tool results concise. Large results eat your context budget. Return summaries, not dumps.

Chapter Challenge

The Remembering Agent. Take the agent loop from Chapter 3 and add the two memory tools from this chapter: remember_fact and recall_facts. Run this two-session test:

Session 1: Tell the agent your name, your favourite programming language, and that you're learning about AI agents.
Session 2: Start a fresh conversation (clear the messages list). Ask: "What do you know about me?"

The agent should recall your facts from the store — even though the messages list is empty. Then add the summarisation strategy: have a long conversation (20+ turns of chitchat), and watch the summariser compress the old turns so the context stays manageable. You've now built an agent with both a past and a manageable context.