Alignment
The degree to which a model acts in line with human values and the goals and intent of the people directing it.
There are two levels to alignment: Broad: does the model act in line with human values and the intent of an org? Everyday: does this system actually do what you want? Labs train models for alignment, but the useful question is always: aligned to whom?
Related
- Sycophancy A model's tendency to tell you what you seem to want to hear.
- Reward hacking A failure where a model or agent games how it is measured, getting a high score without the intended outcome, such as editing tests so they pass instead of fixing the code.
How it connects
- Alignment underpins AI safety
- Alignment is trained in Post-training