Benchmark
A fixed, standardized eval suite held constant so comparisons are fair: across models, across approaches, or over time.
Benchmarks are most commonly thought of as public test sets kept by research groups and AI labs. Your company can and should have its own private benchmarks, built around the tasks you need done, and run evals against them when something new is released or updated. Private benchmarks also avoid a known problem with public ones: the test questions may have leaked into a model's training data.
Kind of
- Eval A measurement of how well an AI system does something (tone, helpfulness, accuracy, safety), usually as a score, graded by code, a model or a person.
Often confused with
- Verified test set A set of test cases that have had correct answers verified by a subject-matter expert.
Related
- Jagged frontier A pattern where a model does some tasks well and fails at others, and the line between them is uneven and hard to predict, even for tasks that seem equally hard.