LLM-as-judge
An eval where a model grades the work of another model or system.
LLM-as-judge is an efficient way to evaluate quality but can be fraught with misalignment to your own judgment. Judges tend to favor longer answers, the first answer shown, and output from their own model family. It requires careful attention to alignment and appropriate instruction before you should have confidence in its judgment.
Also called
- LLM judge
- AI judge
- Model-graded eval
Kind of
- Eval A measurement of how well an AI system does something (tone, helpfulness, accuracy, safety), usually as a score, graded by code, a model or a person.
Related
- Probabilistic guardrail A guardrail that uses a model or classifier, so its judgment can be wrong: it can miss bad content, block good content, or be tricked.
How it connects
- Human-as-judge calibrates LLM-as-judge