Regression eval
An eval designed to protect a working AI feature against unintended change.
Classic software: a regression test, a deterministic check that known-good output stays the same. AI: checking a working feature still performs as well after a change, including the vendor updating or retiring the model. Regression eval: Example: a working chatbot whose tools or documents are rewritten to be more efficient; you need to confirm it works as well as before. Because the output isn't deterministic, exact-match tests aren't enough for the content, so you use eval machinery: a frozen test set, scored, with a threshold you don't drop below, run before and after.
Also called
- Regression test
Kind of
- Eval A measurement of how well an AI system does something (tone, helpfulness, accuracy, safety), usually as a score, graded by code, a model or a person.
Related
- Verified test set A set of test cases that have had correct answers verified by a subject-matter expert.
- Reward hacking A failure where a model or agent games how it is measured, getting a high score without the intended outcome, such as editing tests so they pass instead of fixing the code.
How it connects
- Deterministic alone is not enough for Regression eval
- Model-agnostic relies on Regression eval