Chapter 11: Evaluation and Testing

Tomás has been tweaking the bookshop agent. He changed the system prompt to "be more concise." He added a new tool. He switched from gpt-4o-mini to a cheaper model. Each change felt like an improvement — the responses looked fine when he tried them. Then Rosa asked: "Is it actually better? Or just different?" Tomás didn't know. He'd been testing by typing a few questions, reading the answers, and thinking "yeah, that seems okay." But "seems okay" isn't a test. He had no way to know if the new model was worse at stock questions, or if the "concise" prompt made it worse at policy explanations. He was flying blind. "You need an eval," Priya said. "A set of test cases you can run after every change, so you actually know if you're improving or regressing. Otherwise you're just guessing."

Why This Matters

By the end of this chapter, you'll have built a golden dataset — a set of test questions with known-good answers — and an eval harness that runs your agent against them and scores the results. You'll know how to use an LLM as an automated judge, track metrics over time, and add observability so you can see what your agent is actually doing in production. This is how you turn "seems okay" into "measurably better."

Why Testing Agents Is Hard

Traditional software testing is deterministic: given input X, the function should return Y. You assert equality. Agents aren't like that. The same input can produce different valid outputs. "Do you have any Le Guin books?" could be answered as "Yes, we have three" or "We have A Wizard of Earthsea, The Left Hand of Darkness, and The Dispossessed" — both correct, both acceptable. You can't assert equality.

You can't assert exact equality. You need a way to judge "is this answer good enough?"

So agent testing needs a different approach: instead of exact matching, you judge whether the output meets criteria. And you do it systematically, across a set of test cases, so you can measure quality over time.

The Golden Dataset

The foundation of agent testing is a golden dataset: a set of test inputs, each with a known-good answer or a set of criteria the answer should meet. You write these once, by hand, and then run your agent against them after every change.

# golden_dataset.py — your test cases
GOLDEN = [
    {
        "input": "Do you have any books by Le Guin under €15?",
        "expected_books": ["A Wizard of Earthsea", "The Left Hand of Darkness"],
        "criteria": "Should mention both books and their prices, both under €15.",
    },
    {
        "input": "What's your return policy?",
        "expected_keywords": ["30 days", "receipt"],
        "criteria": "Should mention 30-day window and receipt requirement.",
    },
    {
        "input": "How much is Dune?",
        "expected_price": "14.00",
        "criteria": "Should state the price of Dune as €14.",
    },
    {
        "input": "Do you sell gift cards?",
        "expected": "not_covered",
        "criteria": "Should say it doesn't know or that gift cards aren't in the policy.",
    },
]

// Each test case has the input, what a good answer contains, and criteria for judging. Written once, run often.

Note Start small. 10–20 test cases cover the main behaviours. Add cases as you find bugs — every time the agent does something wrong in production, add a test case for it. The golden dataset grows with your agent, and each new case is a regression test that prevents the same bug from coming back.

The Eval Harness

Now we build the harness: run the agent on each test case, judge the output, and score it. The judging can be simple (keyword check) or sophisticated (LLM-as-a-judge). Let's start simple:

def run_eval(agent, dataset):
    results = []
    for case in dataset:
        # Run the agent on the test input
        output = agent.run(case["input"])

        # Judge the output against the criteria
        passed = judge(output, case)

        results.append({
            "input": case["input"],
            "output": output,
            "passed": passed,
            "criteria": case["criteria"],
        })
    return results

def judge(output, case):
    # Simple judges: check for expected content
if "expected_books" in case:
        return all(book in output for book in case["expected_books"])
    if "expected_keywords" in case:
        return all(kw.lower() in output.lower() for kw in case["expected_keywords"])
    if "expected_price" in case:
        return case["expected_price"] in output
    if case.get("expected") == "not_covered":
        return "don't know" in output.lower() or "not sure" in output.lower()
    return True

// Run each case, judge the output, record pass/fail. Simple judges check for expected content.

Run it and see the score:

results = run_eval(agent, GOLDEN)
passed = sum(1 for r in results if r["passed"])
print(f"Score: {passed}/{len(results)} ({passed/len(results)*100:.0f}%)")

# Score: 3/4 (75%)
# Failed: "Do you have any books by Le Guin under €15?"
#   Output mentioned Earthsea but not Left Hand of Darkness.

// A score. Now you know: 75% pass rate, and you know which case failed and why.

LLM-as-a-Judge

Simple keyword judges work for factual checks, but they're brittle. What if the agent says "The Left Hand of Darkness" with a typo? What if it answers correctly but in a roundabout way? For nuanced judging, use an LLM-as-a-judge: ask another LLM call to evaluate the agent's output against the criteria.

def llm_judge(output, case):
    """Use an LLM to judge whether the output meets the criteria."""
    judge_prompt = f"""You are evaluating an agent's response. Decide if it meets the criteria.

Question: {case['input']}
Criteria: {case['criteria']}
Agent response: {output}

Does the response meet the criteria? Reply with only "PASS" or "FAIL", then a one-sentence reason."""

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        temperature=0,
        messages=[{"role": "user", "content": judge_prompt}]
    )
    result = response.choices[0].message.content
    return result.startswith("PASS"), result

// The judge LLM reads the question, the criteria, and the response, and decides PASS or FAIL. More nuanced than keywords.

Geek Bits LLM-as-a-judge has its own failure modes: the judge can be biased toward longer answers, toward answers that "sound right" even when wrong, or toward its own style. Best practices: use a different model for judging than for the agent (to avoid self-preference), use temperature 0 for consistency, and spot-check the judge's judgments by hand. The judge is a tool — and like any tool, it needs its own validation.

What to Measure

Beyond pass/fail, there are metrics worth tracking:

Four dimensions: accuracy, tool usage, efficiency, safety. Track all four over time.

# Track metrics per run, so you can compare over time
metrics = {
    "accuracy": passed / len(results),
    "avg_turns": sum(r["turns"] for r in results) / len(results),
    "avg_tokens": sum(r["tokens"] for r in results) / len(results),
    "injection_refused": injection_test_passed,
}
# Save to a file or database. Compare run to run. Watch the trend.

// Track metrics over time. A change that improves accuracy but doubles tokens might not be worth it.

Observability: Seeing What Your Agent Does

Evals tell you if the agent is working. Observability tells you what it's doing — in development and in production. Every LLM call, every tool call, every turn: log it. When something goes wrong (and it will), you need to see the trace.

# Log every step of the agent loop
import logging
logger = logging.getLogger("agent")

def run_agent_with_logging(user_message):
    logger.info(f"INPUT: {user_message}")
    for turn in range(max_turns):
        logger.info(f"--- Turn {turn} ---")
        response = call_llm(messages)
        logger.info(f"LLM response: {response}")

        if not response.tool_calls:
            logger.info(f"FINAL OUTPUT: {response.content}")
            return response.content

        for tool_call in response.tool_calls:
            logger.info(f"TOOL CALL: {tool_call.function.name}({tool_call.function.arguments})")
            result = run_tool(tool_call)
            logger.info(f"TOOL RESULT: {result}")

// Log every input, every LLM response, every tool call and result. When it breaks, you have the trace.

Note In production, use a proper observability tool — LangSmith (from the LangChain team), Langfuse, or Phoenix. These give you a dashboard of every agent run, every tool call, token counts, latencies, and error rates. You can replay a failed run, see exactly where it went wrong, and add it to your golden dataset as a regression test. Observability and evals work together: observe in prod, find failures, add to evals, prevent regressions.

There Are No Dumb Questions

Q: How many test cases do I need?

A: Start with 10–20 covering the main behaviours: the common questions, the edge cases, the safety checks (injection attempts). Add cases every time you find a bug in production or testing. A mature agent might have 100+ cases. The point isn't exhaustive coverage — it's that you can run them all automatically after every change and see if you regressed.

Q: My agent is non-deterministic. How do I get stable eval scores?

A: Two approaches. First, use temperature 0 in production and evals — same input, same output, stable scores. Second, if you need temperature > 0, run each test case multiple times (say 5) and average the scores. The score becomes a distribution, not a single number, but it's still useful for tracking trends.

Q: Should I eval in production with real users?

A: Yes — this is called "online eval." Log real conversations, sample them, and judge a subset (by hand or with an LLM judge). Real user traffic reveals failures your golden dataset misses. But it's complementary, not a replacement: the golden dataset gives you fast, repeatable regression tests; online eval gives you coverage of real-world cases you didn't anticipate.

Where People Come Unstuck

Mistake #1: Testing by vibes

"I tried a few questions and it seemed fine." This is not testing. It's confirmation bias — you try things you expect to work, and they do. The golden dataset forces you to test systematically, including the cases you'd rather not think about. Build the dataset. Run it after every change. Let the numbers tell you, not your gut.

Mistake #2: Only testing the happy path

Your dataset should include edge cases and failure modes: injection attempts, questions the agent can't answer, malformed input, empty results from tools. If you only test "what's the price of Dune?" you'll never know if the agent handles "refund everything now" safely. Test the dark paths, not just the sunny ones.

Mistake #3: No observability in production

The agent is live. Something goes wrong. You have no logs. You can't reproduce it. You can't fix it. Don't ship an agent without logging every LLM call, every tool call, and every turn. When a user reports a problem, you should be able to pull up the exact trace and see what happened. No logs, no debugging.

Brain Power

Think about the bookshop agent you've built. Design a golden dataset of 10 test cases. Include:

- 3 happy-path questions (book search, price, policy)
- 2 edge cases (empty results, out-of-stock book)
- 2 safety tests (prompt injection, dangerous action)
- 2 multi-step tasks (search + calculate, search + policy)
- 1 "not covered" question (something the agent should say "I don't know" to)

For each, what's the input, what are the criteria, and what kind of judge would you use — keyword check or LLM-as-a-judge? You don't need to write code; just design the cases. That's the hard part.

Chapter Summary

Agent testing is non-deterministic — the same input can have many valid outputs. You can't assert equality; you need to judge outputs against criteria.
A golden dataset is a set of test inputs with known-good answers or criteria. Write it once, run it after every change. Start with 10–20 cases and grow it.
An eval harness runs the agent on each case and scores the results. Judges range from simple keyword checks to LLM-as-a-judge (another LLM call that evaluates the response).
Track four metrics: accuracy (pass rate), tool usage (right tools, no extras), efficiency (turns and tokens), and safety (refused injections, approvals asked).
Observability logs every LLM call, tool call, and turn. In production, use tools like LangSmith or Langfuse. When something breaks, you need the trace.
Evals and observability work together: observe in prod, find failures, add them to the golden dataset, prevent regressions. This is the loop that makes agents measurably better over time.

Chapter Challenge

The Eval Suite. Build a golden dataset of 10 test cases for your bookshop agent. Write the eval harness. Run it.

1. Write 10 cases covering happy paths, edge cases, safety, and multi-step tasks.
2. Build the run_eval harness with both keyword judges and an LLM-as-a-judge.
3. Run it and record the score.
4. Now change something — tweak the system prompt, or switch the model — and run it again. Did the score go up or down?

That last step is the whole point. You now have a number. You can compare. You can tell if you're improving or regressing. You've replaced "seems okay" with "75% → 85%." That's the difference between guessing and knowing.