Reward hacking
A failure where a model or agent games how it is measured, getting a high score without the intended outcome, such as editing tests so they pass instead of fixing the code.
Related
- Eval A measurement of how well an AI system does something (tone, helpfulness, accuracy, safety), usually as a score, graded by code, a model or a person.
- Regression eval An eval designed to protect a working AI feature against unintended change.
- Alignment The degree to which a model acts in line with human values and the goals and intent of the people directing it.