Production monitoring
The practice of running evals on a live system to score the quality of its outputs.
This type of eval is useful feedback for the owner of the product or system. People use the scores to improve prompts, data or models.
Also called
- Online evals
- Live monitoring
Kind of
- Eval A measurement of how well an AI system does something (tone, helpfulness, accuracy, safety), usually as a score, graded by code, a model or a person.
Often confused with
- Observability The tracking and recording of what an AI system does on each run: inputs, outputs, steps, tool calls, cost and errors.
Related
- Production The state of a process or product that real work depends on and expects to be reliable.