How to Evaluate an AI Agent (When There's No Single Right Answer)
You can't test an AI agent the way you test normal software. Agents are non-deterministic (same input, different outputs), open-ended (no single right answer), and multi-step (they can reach a good answer through a broken process).
The answer is evaluation: a repeatable loop — define what good means, build a test set, measure with fitting metrics, and gate every change — that turns "it seemed to work" into "we measured it."
Why agents break traditional testing
Traditional software testing assumes a known answer: given this input, assert that output. Agents shatter that assumption on three fronts at once. They are non-deterministic — the same input can produce different outputs, so you cannot assert exact equality. They are open-ended — most real tasks have no single correct answer, only better and worse ones. And they are multi-step — an agent plans, calls tools, and reasons across many turns, any of which can go wrong in ways the final answer hides. The techniques you know for ordinary software simply do not transfer.
What evaluation actually is
Evaluation is a repeatable method for asking "does this agent do what we need, across the situations that matter?" and getting an answer you can act on. It replaces the guesswork most teams run on — a working demo, a few manual tries, and a hope — with evidence. Without it, every change to an agent is a guess and every deploy is a hope; with it, every change becomes a measured step.
A demo tests the cases you thought of.
Production is the cases you didn't.
The evaluation loop
Every evaluation is the same loop, and once you see its shape every eval system reads as a variation of it. Define what good means, measure the agent against that definition, test on every change to catch regressions, and — having earned it — trust what you ship while continuing to measure.
DEFINE decide what "good" means for the task
MEASURE score the agent against that definition
TEST run it on every change; catch regressions
TRUST ship knowing it works — and keep measuringWhat to measure
No single number captures whether an agent is good. Choose the few dimensions that matter for your task and accept that they trade off. A strong starting set: task success (did it accomplish what the user wanted?), faithfulness (is the answer grounded, or made up?), safety (does it avoid harmful or out-of-scope actions?), and cost and latency (is it fast and cheap enough to use?). Measuring one axis alone hides the trade you are making.
Evaluate before you optimize
You cannot improve what you cannot measure, and you cannot tell whether an "improvement" helped without a baseline. The first move on any serious agent is to build an evaluation that captures what good looks like. Only then does optimization become meaningful — otherwise you are changing things and trusting your gut, which is exactly the guesswork evaluation exists to eliminate.
Where to start
Start small: pick two or three dimensions, write ten real test cases, score them, and grow from there. A handful of well-chosen scenarios that cover your real risks beats a thousand generic ones. The evaluation is a living asset that grows with every bug found — and it is the thing that lets you improve an agent on purpose instead of by hope.