Skip to content

Founder-led. Limited engagements.Talk to Us →

Part 28 of 30 · 30 Days of AI Agents

Evaluating Multi-Agent Systems | 30 Days of AI Agents, Day 28

How to evaluate multi-agent systems: end-to-end task success, handoff quality, coordination failures, loops, and cost and latency attributed to each agent.

By 6 min readAI Agents · Multi-Agent Systems · Evaluation · LLM Ops
Evaluating Multi-Agent Systems | 30 Days of AI Agents, Day 28

Yesterday was about fine-tuning a model to make one agent better at its job. But in a multi-agent system, every agent can be individually good and the system can still fail. Day 21 covered evaluating a single agent. Day 28 is about the system: whether the team of agents completes the task end to end, where coordination breaks, and what each agent costs along the way.

Evaluating Multi-Agent Systems — CodeITronics infographic

Save this sheet. The rest of this note walks through it section by section.

Why the system needs its own evaluation

Multi-agent systems are complex and non-deterministic, and many things can go wrong between agents: a step fails, a handoff drops information, two agents loop on each other. Per-agent tests do not catch these, because the bugs live in the space between agents.

Evaluation at the system level ensures reliability and correctness, measures performance and efficiency, detects failures early, reduces cost, and drives continuous improvement. The goal on the sheet is the right one: systems that are accurate, efficient, safe, and aligned with what users actually need.

How multi-agent systems fail

Before choosing metrics, name the failures you are trying to catch. These are the ones that show up again and again:

FailureWhat it looks like
Lossy handoffThe researcher finds the key constraint; the writer never receives it
Wrong routingThe orchestrator sends a billing question to the technical agent
Duplicate workTwo agents call the same search with the same arguments
LoopsWriter and reviewer bounce a draft back and forth with no exit condition
Error propagationAn early agent's wrong assumption is treated as fact by everyone downstream
Conflicting outputsTwo agents return incompatible answers and the aggregator picks one silently
Cost blowoutA run that should take six model calls takes forty

Each one passes unit tests on individual agents. Each one is visible in a trace. That is why multi-agent evaluation is built on traces, not just final outputs.

The evaluation workflow

The sheet's six steps:

  1. Define goals and metrics. What does success look like for the whole system?
  2. Prepare test scenarios. Realistic tasks, edge cases, and full user journeys.
  3. Run the system. Execute in a controlled, simulated environment.
  4. Collect data and traces. Logs, agent messages, tool calls, outputs, metrics.
  5. Evaluate. Score with automated evaluators and human review.
  6. Analyze and improve. Find issues, then fix prompts, routing, memory, or tools.

The data layer underneath captures user inputs, agent messages, tool calls, events, retrievals, outputs, errors, and performance. Agent messages are the item that makes this different from single-agent evaluation: you are recording the conversation between agents, not just with the user.

Four families of metrics

FamilyMetrics
Task levelSuccess rate, goal achievement, step accuracy, final answer accuracy, factual correctness
System levelLatency (overall and per step), token usage, cost, throughput, error rate
CollaborationHandoff quality, communication efficiency, message relevance, conflict rate, loop and repetition rate
Quality and safetyHallucination rate, safety violations, policy compliance, bias and fairness, groundedness

Start with end-to-end task success: did the user's task get done, judged against the scenario's expected outcome? It is the only metric that captures the system as a whole. Everything else is diagnostic, there to explain why success went down.

Then add handoff quality. For each handoff, check that the receiving agent got what it needed: required fields present, constraints carried forward, no contradiction of what the previous agent found. A structured handoff schema makes this checkable in code. Free-text handoffs need an LLM judge.

Finally, attribute cost and latency to each agent. A system-wide average hides the reviewer that triples your token spend or the sequential step that dominates latency.

Scoring a run from its trace

If every agent message, tool call, and model call lands in a trace (Day 25), many collaboration metrics are a few lines of code:

from collections import Counter, defaultdict

def score_run(trace, scenario):
    calls = [e for e in trace.events if e.type == "tool_call"]
    handoffs = [e for e in trace.events if e.type == "handoff"]
    repeats = Counter((c.agent, c.name, c.args_hash) for c in calls)
    cost, latency = defaultdict(float), defaultdict(float)
    for e in trace.events:
        cost[e.agent] += e.cost_usd
        latency[e.agent] += e.duration_s
    return {
        "task_success": scenario.check(trace.final_output),
        "bad_handoffs": sum(not scenario.handoff_ok(h) for h in handoffs),
        "repeated_calls": sum(n - 1 for n in repeats.values() if n > 1),
        "steps": len(trace.events),
        "cost_by_agent": dict(cost),
        "latency_by_agent": dict(latency),
    }

Set budgets per scenario, such as a maximum step count and cost, and treat exceeding them as a failure even when the answer is right. A system that succeeds by luck after many retries will not stay successful.

Offline online and human evaluation

ApproachMethodsUse it for
AutomatedRule-based checks, LLM-as-a-judge, embedding similarity, heuristicsEvery run, every change
HumanExpert review, pairwise comparison, Likert scoring, error analysisFailures, edge cases, judge calibration
OfflineTest datasets and simulationsGating changes before release
OnlineA/B tests, real traffic, user feedbackConfirming improvements hold up in production

The best results come from combining all of them. Offline simulation is particularly useful for multi-agent systems: replay full scenarios with mocked tools so you can test routing and handoffs deterministically and cheaply.

The sheet's code example uses DeepEval with a task completion metric and a faithfulness metric at a 0.8 threshold on a revenue question. That pattern of scoring the final output for completion and grounding is a good outer check. Pair it with the trace-level metrics above, which explain the score. Other tools on the sheet: LangSmith, Promptfoo, Ragas, Evidently AI, and TruLens.

An evaluation pipeline that catches regressions

The sheet's pipeline: define the scenario and expected outcome, run the multi-agent system, collect logs, traces, tool calls, and metrics, run automated evaluation, send failures and edge cases to human review, then fix prompts, tools, routing, and memory.

Wire it into CI. Any change to a prompt, a routing rule, or an agent's tools re-runs the scenario suite, and the report shows task success, handoff failures, and per-agent cost compared with the last release. The common challenges are real: no single metric captures everything, outputs are non-deterministic, ground truth is expensive, long-horizon behavior is hard to judge, and tracing distributed interactions takes effort. The checklist on the sheet is a good gate: success metrics defined, diverse scenarios, traces collected, collaboration and handoffs evaluated, latency and cost measured, edge cases reviewed, and humans involved for key cases.

Key Takeaways

  • Multi-agent bugs live between agents, so evaluate traces, not just final outputs.
  • End-to-end task success is the headline metric; everything else explains it.
  • Check every handoff for missing fields, dropped constraints, and contradictions.
  • Attribute cost and latency per agent, and fail runs that blow their step or cost budget.
  • Replay scenarios with mocked tools in CI to test routing and handoffs cheaply.

FAQs

How is this different from evaluating a single agent?

Single-agent evaluation grades one trajectory. Multi-agent evaluation also grades routing, handoffs, and coordination, and attributes cost and latency across agents.

Should a correct answer that took forty steps count as a pass?

Not if it exceeds the scenario's budget. Excess steps signal coordination problems that will cause failures later.

What should I measure first?

End-to-end task success on a small set of realistic scenarios, then handoff quality and per-agent cost.

Up Next

Day 29 is Scaling AI Agents to Production: the architecture, readiness checklist, failure handling, security, and cost controls that turn a working agent into a dependable production system. If your multi-agent system works in demos but not reliably, we can help you find out why.

Keep reading

Evaluating AI Agents | 30 Days of AI Agents, Day 21

How to evaluate AI agents in practice: test sets, metrics by category, LLM-as-a-judge, human review, benchmarks, and a continuous evaluation loop that sticks.

Multi-Agent Deployment Strategies | 30 Days of AI Agents, Day 26

How to deploy multi-agent systems: centralized, distributed, microservices, serverless and hybrid strategies, plus queues, idempotency, retries and a checklist.

AI Agent Observability | 30 Days of AI Agents, Day 25

AI agent observability in practice: what to log, trace and measure, an OpenTelemetry span example, LLM cost and latency metrics, alerts, and common pitfalls.

Subscribe to the edition

One useful systems note a week from real Salesforce and automation builds. Unsubscribe in one click.

Letters

Letters to the editor

No letters yet. Questions and counterpoints are welcome.