Red teaming
Testing which imitates an adversarial attack on an AI system in order to learn more about the safety and security of the system.
Red teaming is not a new concept, but it has some unique applications for AI systems. They can be attacked through plain language (prompt injection, jailbreaks), and they can produce harmful output with no attacker at all, so red teams test for both.
Also called
- AI red teaming
- Adversarial testing
Related
- Eval A measurement of how well an AI system does something (tone, helpfulness, accuracy, safety), usually as a score, graded by code, a model or a person.
How it connects
- Red teaming tests AI safety
- Red teaming tests AI security