How to Build a Test Set for Your AI Agent

The single most valuable thing you'll build isn't the agent — it's the test set you evaluate it against. It's the ground truth every version is measured on, and it survives model swaps, framework changes, and rewrites.

Start with ten real cases, each paired with a verdict for what good looks like, and grow the set with every failure you find.

Why the test set is the asset

An evaluation is only as good as the cases it runs. A test set is a collection of scenarios — inputs paired with some notion of what a good response looks like — that represents the situations your agent must handle. It is the ground truth against which every version of the agent is measured, and it is the one asset that survives model changes, framework changes, and rewrites. Build it well and it pays off on every future decision.

Models change. Frameworks change.
The test set endures.

What a good test set contains

• Representative cases — the common situations your agent actually faces, so the score reflects real performance.
• Edge cases — the rare, tricky, and adversarial inputs where agents break, because these are what production surfaces.
• Known failures — every bug you've found, captured as a case, so it can never silently return.
• A verdict per case — an expected answer, a checklist, or a rubric. A case without a verdict can't evaluate anything.

Where test cases come from

The best test cases come from reality. Real user interactions — especially the ones that went wrong — are gold, because they represent situations that actually happen. Every production failure should become a test case. You can supplement with synthetic cases the model or your team generates to cover situations you haven't seen yet, but the core of a strong test set is drawn from real usage, curated over time.

A case is an input plus a verdict

Be precise about what a single case is: an input the agent will receive, and a way to decide whether the response was good. That verdict takes different forms — an exact expected answer, conditions the response must satisfy, a rubric a judge applies, or a reference to compare against. The discipline of writing the verdict for every case is what turns a pile of examples into an actual test set.

{
  "input": "I was charged twice, I want a refund",
  "expects": {
    "task_success": "acknowledges double charge, checks policy",
    "tool": "issue_refund only if within policy",
    "faithfulness": "cites the refund policy",
  },
}

Start small, grow deliberately

A common mistake is waiting to build a huge test set before evaluating anything. Start with ten cases that capture what matters, and grow the set as you learn where the agent fails. Twenty well-chosen scenarios that cover your real risks beat a thousand generic ones. The test set is a living asset that grows with every bug found and every new situation encountered.