technology
AI red teaming: how testers try to break AI models
Also known as red teaming, LLM red teaming, adversarial testing, jailbreak testing
AI red teaming is adversarial testing of an AI system to find vulnerabilities before attackers or users do, and labs call it a critical safety tool.[1] It ranges from expert and crowdsourced human testing to models that attack other models,[2][3] but there is no standard practice,[4] and the UK AI Security Institute has found universal jailbreaks for every system it has tested.[5]
Key facts
What it is
Anthropic describes red teaming as adversarially testing a technological system to identify potential vulnerabilities.[1] In early studies of language models, red teamers looked for harmful outputs such as offensive language, leaked private training data and personal phone numbers.[6][7] Red teaming is one part of wider safety testing; measuring whether a model has dangerous skills at all is the related job of dangerous-capability-evaluations.
Methods
Anthropic’s 2024 overview sorts the methods into four families: domain experts (for example trust and safety, national security and multilingual testing), automated red teaming with language models, multimodal red teaming, and open-ended crowdsourced or community efforts.[2]
- Human red teams. A 2022 Anthropic study red-teamed models of three sizes and released 38,961 attacks for others to study.[8] OpenAI’s white paper on outside testers says the key design choices are who sits on the red team, what access they get and what guidance they receive.[9]
- Automated red teams. Hand-written test cases are expensive, which limits how many and how varied they can be.[10] A 2022 DeepMind paper therefore used one model to attack another and found tens of thousands of offensive replies.[3]
- Red-teaming safeguards. Testers also attack the defences around a model. In over 3,000 estimated hours of red teaming, no tester found a universal jailbreak against an early version of Anthropic’s classifier-guarded model.[11] In February 2026 the UK AI Security Institute said its Boundary Point Jailbreaking method was, it believed, the first automated attack to beat those classifiers.[12]
What red teams have found
Results vary by model and training. In the 2022 Anthropic study, models trained with reinforcement learning from human feedback got harder to red team as they grew, while other model types showed a flat trend.[13] The UK AI Security Institute has found universal jailbreaks for every system it tested, but in one biological-misuse comparison the expert effort needed rose about 40-fold between two models released six months apart.[5]
In July 2026 the institute described a new Control Red Team that attacks the monitors that watch AI agents, working with Google DeepMind and Anthropic.[14] It found vulnerabilities in every version of Anthropic’s agentic coding monitor it tested, and its best attacks informed Anthropic’s testing of Claude Mythos Preview.[15]
Lessons from practitioners
Microsoft’s AI red team drew eight lessons from more than 100 generative AI products, including that red teaming is not the same as safety benchmarking and that the work of securing AI systems will never be complete.[16] It also found that attackers do not need sophisticated gradient-based methods to break a system, and that automation helps coverage while the human element stays crucial.[17]
Limits
There is no agreed method. Anthropic wrote in 2024 that developers may use different techniques to test the same threat, which makes results hard to compare.[4] OpenAI’s white paper also sets out the limits of external red teaming and places it inside a broader set of evaluations.[18] And because no single safeguard holds up on its own, the International AI Safety Report 2026 recommends layering several defences.[19]
Read next
Dangerous capability evaluations: testing AI for severe risks
Dangerous capability evaluations test whether AI models can help with cyberattacks, weapons or self-replication. How they work and their limits.
Questions readers ask
What is AI red teaming?
Adversarially testing an AI system to find its weaknesses, such as prompts that make it produce harmful content or bypass its safeguards. Anthropic calls it a critical tool for AI safety and security.[1]
Is red teaming the same as a safety benchmark?
No. Microsoft's AI red team, drawing on more than 100 products it has tested, lists "AI red teaming is not safety benchmarking" among its main lessons.[16]
Can AI models red-team other AI models?
Yes. A 2022 DeepMind paper used one language model to generate test cases for another and found tens of thousands of offensive replies from a 280-billion-parameter chatbot.[3]
Can safeguards survive red teaming?
Not fully so far. The UK AI Security Institute found universal jailbreaks for every system it tested, although the effort needed rose sharply between two models in one comparison.[5]
Does red teaming get harder as models improve?
In one 2022 study, models trained with reinforcement learning from human feedback became harder to red team as they grew larger, while other model types showed no trend with size.[13]
Sources
· 19 claims from 8 publishers
Each numbered claim is a statement we checked against the sources listed with it. Tap a number in the text to see its sources in place.
- [1]
Anthropic describes red teaming as adversarially testing a technological system to identify potential vulnerabilities, and calls it a critical tool for improving the safety and security of AI systems. confirmedas of 2024-06-12
- Challenges in red teaming AI systems · Anthropic · Primary source · 2024-06-12 (retrieved 2026-11-14)
- [2]
Anthropic's 2024 overview groups red teaming methods into domain-specific expert teaming (such as trust and safety, national security and multilingual testing), automated red teaming with language models, multimodal red teaming, and open-ended crowdsourced or community red teaming. confirmedas of 2024-06-12
- Challenges in red teaming AI systems · Anthropic · Primary source · 2024-06-12 (retrieved 2026-11-14)
- [3]
A February 2022 DeepMind paper used one language model to generate test cases for another, uncovering tens of thousands of offensive replies from a 280-billion-parameter chatbot. confirmedas of 2022-02-07
- Red Teaming Language Models with Language Models · arXiv (DeepMind) · Primary source · 2022-02-07 (retrieved 2026-11-14)
- [4]
Anthropic wrote in June 2024 that the lack of standardized practices for AI red teaming means developers may use different techniques to assess the same threat, making results hard to compare. confirmedas of 2024-06-12
- Challenges in red teaming AI systems · Anthropic · Primary source · 2024-06-12 (retrieved 2026-11-14)
- [5]
AISI found universal jailbreaks for every system it had tested, but in one biological-misuse comparison the expert effort needed rose about 40-fold (from about 10 minutes to about 7 hours) between two models released six months apart. confirmedas of 2025-12-18
- Frontier AI Trends Report · UK AI Security Institute · Primary source · 2025-12-18 (retrieved 2026-10-10)
- [6]
The 2022 Anthropic red teaming study found a variety of harmful outputs, ranging from offensive language to more subtly harmful non-violent unethical outputs. confirmedas of 2022-11-22
- Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned · arXiv (Anthropic) · Primary source · 2022-08-23 (retrieved 2026-11-14)
- [7]
The 2022 DeepMind automated red teaming paper also found personal and hospital phone numbers generated as the chatbot's own contact information and leakage of private training data. confirmedas of 2022-02-07
- Red Teaming Language Models with Language Models · arXiv (DeepMind) · Primary source · 2022-02-07 (retrieved 2026-11-14)
- [8]
A 2022 Anthropic study red-teamed language models of 2.7 billion, 13 billion and 52 billion parameters and released a dataset of 38,961 red team attacks. confirmedas of 2022-11-22
- Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned · arXiv (Anthropic) · Primary source · 2022-08-23 (retrieved 2026-11-14)
- [9]
An OpenAI white paper on external red teaming says the design choices include the composition of the red team, the access levels given and the guidance provided, and that red teaming can feed risk assessments and automated evaluations. confirmedas of 2025-01-24
- OpenAI's Approach to External Red Teaming for AI Models and Systems · arXiv (OpenAI) · Primary source · 2025-01-24 (retrieved 2026-11-14)
- [10]
The 2022 DeepMind paper noted that hand-written test cases from human annotators are expensive, which limits the number and diversity of tests. confirmedas of 2022-02-07
- Red Teaming Language Models with Language Models · arXiv (DeepMind) · Primary source · 2022-02-07 (retrieved 2026-11-14)
- [11]
In over 3,000 estimated hours of red teaming, no red teamer found a universal jailbreak against an early classifier-guarded model at comparable detail, and the classifiers added 0.38 percentage points to production refusals with a 23.7% inference overhead. confirmedas of 2025-01-31
- Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming · arXiv (Sharma et al., Anthropic) · Primary source · 2025-01-31 (retrieved 2026-10-10)
- [12]
In February 2026 AISI said its Boundary Point Jailbreaking method was, it believed, the first automated attack to succeed against Anthropic's Constitutional Classifiers. confirmedas of 2026-02-17
- Boundary Point Jailbreaking: A new way to break the strongest AI defences · UK AI Security Institute · Primary source · 2026-02-17 (retrieved 2026-10-10)
- [13]
The 2022 Anthropic study found that models trained with reinforcement learning from human feedback became harder to red team as they grew larger, while other model types showed a flat trend with scale. confirmedas of 2022-11-22
- Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned · arXiv (Anthropic) · Primary source · 2022-08-23 (retrieved 2026-11-14)
- [14]
In July 2026 AISI described a new Control Red Team that stress-tests the monitors frontier developers use to watch AI agents, having tested an asynchronous reasoning monitor with Google DeepMind and successive versions of an agentic coding monitor with Anthropic. confirmedas of 2026-07-23
- How our Control Red Team is stress-testing frontier monitors · UK AI Security Institute · Primary source · 2026-07-23 (retrieved 2026-10-10)
- [15]
AISI reported finding vulnerabilities in every version of Anthropic's agentic coding monitor that it tested, with its best attacks informing Anthropic's testing of Claude Mythos Preview. confirmedas of 2026-07-23
- How our Control Red Team is stress-testing frontier monitors · UK AI Security Institute · Primary source · 2026-07-23 (retrieved 2026-10-10)
- [16]
Microsoft's AI red team published eight lessons in January 2025 from red teaming more than 100 generative AI products, including that AI red teaming is not safety benchmarking and that securing AI systems will never be complete. confirmedas of 2025-01-13
- Lessons From Red Teaming 100 Generative AI Products · arXiv (Microsoft) · Primary source · 2025-01-13 (retrieved 2026-11-14)
- [17]
Among Microsoft's lessons are that attackers do not need to compute gradients to break an AI system, that automation can cover more of the risk landscape, and that the human element remains crucial. confirmedas of 2025-01-13
- Lessons From Red Teaming 100 Generative AI Products · arXiv (Microsoft) · Primary source · 2025-01-13 (retrieved 2026-11-14)
- [18]
OpenAI's white paper also describes the limitations of external red teaming and how it fits into a broader range of model and system evaluations. confirmedas of 2025-01-24
- OpenAI's Approach to External Red Teaming for AI Models and Systems · arXiv (OpenAI) · Primary source · 2025-01-24 (retrieved 2026-11-14)
- [19]
The International AI Safety Report 2026 says layering multiple safeguards (defence in depth) gives more robust protection than relying on a single intervention. confirmedas of 2026-02-03
- International AI Safety Report 2026 · International AI Safety Report · Primary source · 2026-02-03 (retrieved 2026-10-10)
Revision history (1)
- Page created.
Created Nov 14, 2026. Last reviewed by an editor on Nov 14, 2026. Next scheduled review: Aug 14, 2027.
Cite this page
"AI red teaming: how testers try to break AI models." ContentLora, updated Nov 14, 2026. https://contentlora.com/wiki/ai-red-teaming
Spotted an error? Suggest a correction or emailcorrections@contentlora.com.