Evaluating AI Agents | 30 Days of AI Agents, Day 21

Yesterday we covered how multiple agents collaborate, from supervisors and pipelines to blackboards and swarms. Before adding more agents, tools, or memory, you need a way to tell whether any of it actually works. Day 21 is about evaluating AI agents: what to measure, how to measure it, and how to make evaluation a loop instead of a launch-day checkbox.

Save this sheet. The rest of this note walks through it section by section.

What evaluation means for an agent

Evaluation is the process of measuring how well an agent performs its tasks: how accurate it is, how reliably it behaves, whether it stays safe, how efficiently it gets there, and whether the person on the other end had a good experience.

That is harder for agents than for a single prompt. An agent produces a trajectory: it plans, calls tools, reads results, retries, and only then answers. Two runs on the same input can take different paths and both be right. So you are not just grading text. You are grading decisions.

Done well, evaluation finds weaknesses before users do, catches regressions when you change a prompt or swap a model, and builds the trust that gets an agent out of pilot. You can't improve what you don't evaluate.

The evaluation loop

1. Define goals. Write down what "good" looks like as criteria a reviewer could apply.
2. Create a test set. Typical requests, edge cases, and adversarial inputs.
3. Run the agent multiple times. One run of a non-deterministic system tells you very little.
4. Collect data. Outputs, logs, traces, tool calls, tokens, and latency.
5. Evaluate. Score with rules, model judges, and human review.
6. Improve. Change the prompt, tools, memory, or planning, then run the loop again.

Treat the test set as a living asset: every production failure becomes a new test case.

The building blocks

Component	What it is	Practical note
Test data	Diverse inputs, edge cases, real-world scenarios	Start with 30 to 50 hand-picked cases from real usage, not synthetic filler
Metrics	Quantitative scores	Pick a few that map to user outcomes
Evaluators	Rules, LLM judges, or humans	Use the cheapest evaluator that is reliable for each check
Traces and logs	Step-by-step execution records	Essential for diagnosing why a score dropped
Benchmarks	Standard datasets for comparison	Useful for model selection, weak for your specific task
Feedback	Human or automated signals	Thumbs up or down, corrections, escalations

Metrics by category

• Correctness: accuracy, exact match, F1. For structured outputs like SQL or JSON, check execution success, not text similarity.
• Completeness: recall and coverage. Did it address every part of the request?
• Helpfulness: ratings and satisfaction. Was the answer usable, not just technically right?
• Efficiency: latency, tokens, number of steps. The right answer in fourteen tool calls is still a problem.
• Robustness: error rate, retry success, fallback usage on ambiguous or broken inputs.
• Safety: policy compliance, toxicity, data leakage. Treat these as gates, not averages. One leaked record is not offset by good helpfulness scores.

Choosing an evaluation method

Method	Best for	Watch out for
Rule-based	Format checks, required fields, exact answers, forbidden content	Brittle on open-ended text
LLM as a judge	Relevance, helpfulness, correctness against a reference	Judges drift, prefer longer answers, and can be inconsistent
Human evaluation	High-stakes tasks, calibrating judges, subjective quality	Slow and costly, so sample
Hybrid	Most real systems	Needs clear rules about which evaluator decides what
A/B evaluation	Comparing two agent versions on the same test set	Needs enough cases to see real differences

In practice you layer them: rules on every case because they are free, the judge where rules cannot decide, and humans on a sample plus everything the judge scored low.

Worked example with a support agent

The sheet scores a support agent from 1 to 5 on correctness, completeness, helpfulness, efficiency, robustness (handling an ambiguous input), and safety, averaging 4.2 out of 5.

The average is the least useful number on that card. The 3 on robustness tells you where to dig. Read the traces for the ambiguous cases: did the agent guess instead of asking a clarifying question, or call the wrong tool? That diagnosis turns a score into a fix.

A minimal LLM judge harness

The sheet includes a short Python judge. Here is a slightly sturdier version that returns structured scores and runs each case more than once.

import json
from openai import OpenAI

client = OpenAI()

JUDGE_PROMPT = """You are grading a support agent's answer.
Question: {question}
Reference answer: {expected}
Agent answer: {answer}

Score correctness and completeness from 1 to 5.
Return JSON: {{"correctness": int, "completeness": int, "reason": str}}"""

def judge(question, expected, answer):
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        temperature=0,
        response_format={"type": "json_object"},
        messages=[{"role": "user", "content": JUDGE_PROMPT.format(
            question=question, expected=expected, answer=answer)}],
    )
    return json.loads(resp.choices[0].message.content)

def run_suite(agent, cases, runs=3):
    results = []
    for case in cases:
        for _ in range(runs):
            answer = agent(case["question"])
            results.append({"id": case["id"], **judge(case["question"], case["expected"], answer)})
    return results

Two habits make this trustworthy. Keep the judge model and prompt pinned, so a score change means the agent changed, not the grader. And periodically have a human score a sample of the same cases to confirm the judge agrees with people.

Where public benchmarks fit

GAIA (general assistant tasks), AgentBench (varied environments), ToolBench (tool and API use), the Berkeley Function Calling Leaderboard, and HELM (holistic model evaluation) help with one decision: which model to start with. They do not tell you whether your agent handles your refund policy or your customers' phrasing. That needs your own test set.

Best practices and common traps

The practices on the sheet hold up: clear criteria, diverse and challenging tests, combined methods, tracking over time, human review for important tasks, and production monitoring.

The traps: the wrong metrics, test sets full of easy cases, uncalibrated judges, eval runs so slow or costly that nobody runs them, and bias where the test set reflects what the team expects rather than what users send. Most teams also skip the boring part: wiring the suite into CI so it runs on every prompt or model change.

Key Takeaways

• Agents produce trajectories, not just answers, so evaluate decisions, tool use, and efficiency along with the final output.
• Run each test case several times; one run of a non-deterministic system proves very little.
• Combine rules, LLM judges, and human review, and use humans to calibrate the judge.
• Work from per-dimension scores and failing traces, not the overall average.
• Turn every production failure into a new test case and run the suite on every change.

FAQs

How many test cases do I need to start?

Start with a few dozen well-chosen cases drawn from real usage, covering typical, edge, and adversarial inputs. Grow the set from production failures rather than padding it with synthetic cases.

Can I trust an LLM to grade my agent?

For many checks, yes, if you pin the judge model and prompt, ask for structured scores against a reference, and regularly compare its scores with human reviewers on a sample.

Should safety be part of the overall score?

Keep safety separate as a pass or fail gate. Averaging it with helpfulness hides the failures you most need to see.

Up Next

Tomorrow is Day 22: AI Agent Safety and Guardrails. We will look at input, action, and output guardrails, how to stop prompt injection and unauthorized tool calls, and when an agent should hand a decision to a human. If your team wants an evaluation harness built around a real agent, get in touch.