Guardrail
A check, built into the harness or written into the instructions, that screens what goes in, what comes out, or what an agent tries to do.
Guardrails are an essential component of a robust AI system that has AI safety and AI security protections. Checks that run in code outside the model are much stronger than rules written in the instructions, which a model can ignore and an attacker can override. They can also be an important part of a product experience, ensuring that the system is behaving as intended and steering it back on track if not.
Also called
- Safety filters
- Content filters
- Policy checks
- Input/output validation
Kinds
- Deterministic guardrail A guardrail that uses fixed rules or code so that it applies the same way every time.
- Input guardrail A guardrail that checks or filters what goes into a model before the model processes it.
- Output guardrail A guardrail that checks or filters what comes out of a model before anyone sees it or it takes effect.
- PII redaction A type of guardrail that removes or masks personal data in text passing through a system, usually before it reaches a model or log.
- Probabilistic guardrail A guardrail that uses a model or classifier, so its judgment can be wrong: it can miss bad content, block good content, or be tricked.
Related
- Human oversight Human involvement in an AI process to monitor, govern or stop it, before it acts, after it acts, or only for certain actions.
- Sandbox An isolated environment where an agent can execute code or take actions without accessing live systems or data.
- Least privilege for AI A rule that an AI system or agent gets only the permissions it strictly needs for the task, and nothing more.
How it connects
- Guardrail filter inputs and outputs of Agent
- AI security defended by Guardrail
- AI safety defended by Guardrail