What is LLM-as-a-judge?
LLM-as-a-judge uses a language model to evaluate an agent's output when no automatic check exists. Its strengths, weaknesses and safe use.
Updated
LLM-as-a-judge is the practice of using a language model to evaluate another model's or agent's output against criteria, for example deciding whether an agent run achieved its task, when no automatic check is available.
Strengths and weaknesses
- Strengths: scales to open-ended tasks, needs no hand-written assertions, explains its verdicts.
- Weaknesses: can be fooled by confident claims, varies between runs, shares blind spots with the model it judges, and can be manipulated by injected text in what it reads.
Using a judge well
- Prefer hard signals. A passing test, a business event or a user's rating beats any judge. Use the judge only when none exists.
- Judge the steps, not just the answer. An agent that claims success after a failed key step should be scored as a failure.
- Cap its confidence, and lower it further for runs that saw untrusted input.
- Do not let judge verdicts alone change what an agent relies on without a person or harder evidence.
In Retrack
Retrack's judge runs only when a run has no outcome signal, at most once, with confidence capped at 0.7 (0.5 for untrusted runs). Its verdicts add support to playbooks but never promote one on their own. See Verifying runs.