Side-by-side eval
An eval that compares the outputs of two or more models or systems next to each other to judge which is better.
Blind comparison, where you hide which model produced which output, is one of the most effective ways to remove your personal bias toward certain models. Side-by-side comparisons also show quickly which output fits your preferences.
Also called
- Side-by-side evaluation
- Pairwise comparison
- A/B comparison
Kind of
- Eval A measurement of how well an AI system does something (tone, helpfulness, accuracy, safety), usually as a score, graded by code, a model or a person.
Often confused with
- A/B test An experiment where live users are split between two versions of a product or service and the results are compared.