Yesterday was about fine-tuning a model to make one agent better at its job. But in a multi-agent system, every agent can be individually good and the system can still fail. Day 21 covered evaluating a single agent. Day 28 is about the system: whether the team of agents completes the task end to end, where coordination breaks, and what each agent costs along the way.

Save this sheet. The rest of this note walks through it section by section.
Why the system needs its own evaluation
Multi-agent systems are complex and non-deterministic, and many things can go wrong between agents: a step fails, a handoff drops information, two agents loop on each other. Per-agent tests do not catch these, because the bugs live in the space between agents.
Evaluation at the system level ensures reliability and correctness, measures performance and efficiency, detects failures early, reduces cost, and drives continuous improvement. The goal on the sheet is the right one: systems that are accurate, efficient, safe, and aligned with what users actually need.
How multi-agent systems fail
Before choosing metrics, name the failures you are trying to catch. These are the ones that show up again and again:
| Failure | What it looks like |
|---|---|
| Lossy handoff | The researcher finds the key constraint; the writer never receives it |
| Wrong routing | The orchestrator sends a billing question to the technical agent |
| Duplicate work | Two agents call the same search with the same arguments |
| Loops | Writer and reviewer bounce a draft back and forth with no exit condition |
| Error propagation | An early agent's wrong assumption is treated as fact by everyone downstream |
| Conflicting outputs | Two agents return incompatible answers and the aggregator picks one silently |
| Cost blowout | A run that should take six model calls takes forty |
Each one passes unit tests on individual agents. Each one is visible in a trace. That is why multi-agent evaluation is built on traces, not just final outputs.
The evaluation workflow
The sheet's six steps:
- Define goals and metrics. What does success look like for the whole system?
- Prepare test scenarios. Realistic tasks, edge cases, and full user journeys.
- Run the system. Execute in a controlled, simulated environment.
- Collect data and traces. Logs, agent messages, tool calls, outputs, metrics.
- Evaluate. Score with automated evaluators and human review.
- Analyze and improve. Find issues, then fix prompts, routing, memory, or tools.
The data layer underneath captures user inputs, agent messages, tool calls, events, retrievals, outputs, errors, and performance. Agent messages are the item that makes this different from single-agent evaluation: you are recording the conversation between agents, not just with the user.
Four families of metrics
| Family | Metrics |
|---|---|
| Task level | Success rate, goal achievement, step accuracy, final answer accuracy, factual correctness |
| System level | Latency (overall and per step), token usage, cost, throughput, error rate |
| Collaboration | Handoff quality, communication efficiency, message relevance, conflict rate, loop and repetition rate |
| Quality and safety | Hallucination rate, safety violations, policy compliance, bias and fairness, groundedness |
Start with end-to-end task success: did the user's task get done, judged against the scenario's expected outcome? It is the only metric that captures the system as a whole. Everything else is diagnostic, there to explain why success went down.
Then add handoff quality. For each handoff, check that the receiving agent got what it needed: required fields present, constraints carried forward, no contradiction of what the previous agent found. A structured handoff schema makes this checkable in code. Free-text handoffs need an LLM judge.
Finally, attribute cost and latency to each agent. A system-wide average hides the reviewer that triples your token spend or the sequential step that dominates latency.
Scoring a run from its trace
If every agent message, tool call, and model call lands in a trace (Day 25), many collaboration metrics are a few lines of code:
from collections import Counter, defaultdict
def score_run(trace, scenario):
calls = [e for e in trace.events if e.type == "tool_call"]
handoffs = [e for e in trace.events if e.type == "handoff"]
repeats = Counter((c.agent, c.name, c.args_hash) for c in calls)
cost, latency = defaultdict(float), defaultdict(float)
for e in trace.events:
cost[e.agent] += e.cost_usd
latency[e.agent] += e.duration_s
return {
"task_success": scenario.check(trace.final_output),
"bad_handoffs": sum(not scenario.handoff_ok(h) for h in handoffs),
"repeated_calls": sum(n - 1 for n in repeats.values() if n > 1),
"steps": len(trace.events),
"cost_by_agent": dict(cost),
"latency_by_agent": dict(latency),
}
Set budgets per scenario, such as a maximum step count and cost, and treat exceeding them as a failure even when the answer is right. A system that succeeds by luck after many retries will not stay successful.
Offline online and human evaluation
| Approach | Methods | Use it for |
|---|---|---|
| Automated | Rule-based checks, LLM-as-a-judge, embedding similarity, heuristics | Every run, every change |
| Human | Expert review, pairwise comparison, Likert scoring, error analysis | Failures, edge cases, judge calibration |
| Offline | Test datasets and simulations | Gating changes before release |
| Online | A/B tests, real traffic, user feedback | Confirming improvements hold up in production |
The best results come from combining all of them. Offline simulation is particularly useful for multi-agent systems: replay full scenarios with mocked tools so you can test routing and handoffs deterministically and cheaply.
The sheet's code example uses DeepEval with a task completion metric and a faithfulness metric at a 0.8 threshold on a revenue question. That pattern of scoring the final output for completion and grounding is a good outer check. Pair it with the trace-level metrics above, which explain the score. Other tools on the sheet: LangSmith, Promptfoo, Ragas, Evidently AI, and TruLens.
An evaluation pipeline that catches regressions
The sheet's pipeline: define the scenario and expected outcome, run the multi-agent system, collect logs, traces, tool calls, and metrics, run automated evaluation, send failures and edge cases to human review, then fix prompts, tools, routing, and memory.
Wire it into CI. Any change to a prompt, a routing rule, or an agent's tools re-runs the scenario suite, and the report shows task success, handoff failures, and per-agent cost compared with the last release. The common challenges are real: no single metric captures everything, outputs are non-deterministic, ground truth is expensive, long-horizon behavior is hard to judge, and tracing distributed interactions takes effort. The checklist on the sheet is a good gate: success metrics defined, diverse scenarios, traces collected, collaboration and handoffs evaluated, latency and cost measured, edge cases reviewed, and humans involved for key cases.
Key Takeaways
- Multi-agent bugs live between agents, so evaluate traces, not just final outputs.
- End-to-end task success is the headline metric; everything else explains it.
- Check every handoff for missing fields, dropped constraints, and contradictions.
- Attribute cost and latency per agent, and fail runs that blow their step or cost budget.
- Replay scenarios with mocked tools in CI to test routing and handoffs cheaply.
FAQs
How is this different from evaluating a single agent?
Single-agent evaluation grades one trajectory. Multi-agent evaluation also grades routing, handoffs, and coordination, and attributes cost and latency across agents.
Should a correct answer that took forty steps count as a pass?
Not if it exceeds the scenario's budget. Excess steps signal coordination problems that will cause failures later.
What should I measure first?
End-to-end task success on a small set of realistic scenarios, then handoff quality and per-agent cost.
Up Next
Day 29 is Scaling AI Agents to Production: the architecture, readiness checklist, failure handling, security, and cost controls that turn a working agent into a dependable production system. If your multi-agent system works in demos but not reliably, we can help you find out why.




Letters to the editor
No letters yet. Questions and counterpoints are welcome.