Red Finch Advisory

Eval

A measurement of how well an AI system does something (tone, helpfulness, accuracy, safety), usually as a score, graded by code, a model or a person.

An eval doesn't have to be structured or a comparison: a one-off tone check on a single customer-service transcript is an eval. Not limited to models; you can eval a harness, a context setup, or other AI infrastructure. Comparison and benchmarking are uses of evals, not the definition. Edge case: an eval can have a deterministic answer (did the model solve the math problem?) and still be an eval, because it measures capability.

Also called

  • Evaluations
  • Model evaluation
  • AI testing

Avoid

  • Vibe check
  • Vibes-based eval

Kinds

  • Benchmark A fixed, standardized eval suite held constant so comparisons are fair: across models, across approaches, or over time.
  • LLM-as-judge An eval where a model grades the work of another model or system.
  • Regression eval An eval designed to protect a working AI feature against unintended change.
  • Verified test set A set of test cases that have had correct answers verified by a subject-matter expert.
  • A/B test An experiment where live users are split between two versions of a product or service and the results are compared.
  • Human-as-judge An eval where a person reviews a system's output and judges its quality.
  • Production monitoring The practice of running evals on a live system to score the quality of its outputs.
  • Side-by-side eval An eval that compares the outputs of two or more models or systems next to each other to judge which is better.

Related

  • Red teaming Testing which imitates an adversarial attack on an AI system in order to learn more about the safety and security of the system.
  • Observability The tracking and recording of what an AI system does on each run: inputs, outputs, steps, tool calls, cost and errors.
  • Reward hacking A failure where a model or agent games how it is measured, getting a high score without the intended outcome, such as editing tests so they pass instead of fixing the code.

How it connects

← All terms