Probabilistic guardrail
A guardrail that uses a model or classifier, so its judgment can be wrong: it can miss bad content, block good content, or be tricked.
Also called
- Model-based check
- Classifier
Kind of
- Guardrail A check, built into the harness or written into the instructions, that screens what goes in, what comes out, or what an agent tries to do.
Related
- LLM-as-judge An eval where a model grades the work of another model or system.
- Probabilistic A property of a system or check that works on likelihoods, so the same input(s) can give different outputs, and any single answer can be wrong some of the time.