LLM-as-Judge: How to Use a Model to Evaluate a Model

LLM-as-judge uses a capable model to score or compare agent outputs against a rubric — filling the gap where quality is open-ended and human judgment doesn't scale.

Done well it approximates human judgment cheaply; done carelessly it produces confident nonsense. The keys: a specific rubric, pairwise over absolute scoring, and validating the judge against human labels.

Why use a model as a judge

Many of the qualities that matter most — helpfulness, faithfulness, reasoning quality — have no formula, and human judgment does not scale to thousands of cases on every change. Using a capable model as a judge fills that gap: you ask a model to score or compare outputs against a rubric. Done well it approximates human judgment at a fraction of the cost and effort. Done carelessly it produces confident, systematic nonsense — which is why the details below matter.

Why a model can judge at all

It can seem circular to use a model to evaluate a model — if the judge could reliably tell good from bad, why not use it as the agent? The resolution is that judging is easier than doing. Recognizing whether an answer is faithful to a source is narrower and more constrained than producing the faithful answer, the way it's easier to check a proof than to find one. The judge is handed the input, the output, and a rubric, and asked only to assess against that rubric.

The rubric is everything

The quality of the judgment depends almost entirely on the rubric. A vague instruction to "rate this 1-10" yields noise; a specific rubric that defines each level and what to look for yields something usably consistent. The judge's reliability comes from the structure the rubric provides.

judgment = judge_model(
    task=original_input,
    response=agent_output,
    rubric="""Score faithfulness 1-5.
    5 = every claim supported by the sources.
    3 = mostly supported, minor unsupported detail.
    1 = key claims not supported / contradicted.""",
)

Pairwise beats absolute scoring

Models are more reliable at comparing than at scoring in the abstract. Asking "which of these two responses is better?" tends to be far more consistent than "rate this 1-10," because absolute scores drift and cluster while comparisons are anchored. Whenever you can frame evaluation as a comparison — against a reference, or between two versions of the agent — you get more reliable signal.

Ask which is better, not how good.

The biases to defend against

• Position bias — judges can favor whichever answer comes first; swap the order and average to cancel it.
• Verbosity bias — judges often prefer longer answers regardless of quality; call it out in the rubric.
• Self-preference — a judge may favor outputs from its own model family; be aware when judge and agent share a model.
• Validate the judge — check it against human labels on a sample. An unvalidated judge is an opinion, not a measurement.