Observability
The tracking and recording of what an AI system does on each run: inputs, outputs, steps, tool calls, cost and errors.
Some tools also record a model's visible reasoning. That is often a summary, and it does not always show what drove the answer.
Also called
- Tracing
- AI observability
- LLM monitoring
- Agent tracing
- AgentOps
Kind of
- AI operations A discipline concerned with the AI lifecycle and running AI in production.
Often confused with
- Explainability The ability to understand how or why a model reached a conclusion and produced a given output.
- Production monitoring The practice of running evals on a live system to score the quality of its outputs.
Related
- Eval A measurement of how well an AI system does something (tone, helpfulness, accuracy, safety), usually as a score, graded by code, a model or a person.
- Stack The shared AI infrastructure that every agent and harness in your company draws on, such as gateways, model access, data, evals, central identity and shared connectors.
How it connects
- Observability logs from LLM gateway
- Agent manager watches agents through Observability