Human-as-judge
An eval where a person reviews a system's output and judges its quality.
This can be unavoidable when it comes to niche needs like style or preference. Human grades are the standard other judges are checked against, though people often disagree with each other. Human grades are also the starting point to calibrate an LLM-as-judge.
Also called
- Human evaluation
- Human review
- Human rating
Kind of
- Eval A measurement of how well an AI system does something (tone, helpfulness, accuracy, safety), usually as a score, graded by code, a model or a person.
Often confused with
- Human-in-the-loop A setup where a person approves each action before it happens.
How it connects
- Human-as-judge calibrates LLM-as-judge