Skip to content
ContentLora

    Tip: press / anywhere to search.

    Explainer

    How AI safety testing works: evals, red teams and thresholds

    Before release, frontier AI models are tested for dangerous capabilities such as cyberattacks, bioweapons help and self-replication, and red-teamed for ways around their safeguards, by labs and by government institutes. Results feed company safety frameworks with risk thresholds,[1] but the International AI Safety Report 2026 warns that models telling tests apart from real use has made reliable testing harder.[2]

    Editor reviewedUpdated AI safety and alignmentArtificial intelligence

    What gets tested

    Safety testing asks two questions: what could this model do if someone tried to misuse it, and will its safety rules hold up? Testers focus on a short list of severe risks. The UK AI Security Institute (AISI), for example, measures cyberattack skills, how hard safeguards are to bypass, and whether models can copy themselves.[3][4][5]

    Dangerous-capability evaluations use task suites with human baselines. AISI reported that the best models went from under 9% to about 50% success on apprentice-level cyber tasks between late 2023 and late 2025, with a first expert-level success in 2025,[3] and self-replication task success from 5% to 60% in controlled settings.[5] Autonomy is often summarised as a “time horizon”: METR’s January 2026 update put the doubling time at about 89 days since 2024.[6]

    Red-teaming the safeguards

    Red-teamers try to “jailbreak” a model, tricking it into giving help it should refuse. AISI says it has found a universal jailbreak for every system it has tested.[4]

    Safeguard testing is about cost, not just success: in one biological-misuse comparison, expert effort to jailbreak rose about 40-fold, from about 10 minutes to about 7 hours, between two models released six months apart.[4] This is why the International AI Safety Report 2026 recommends layered defences rather than any single safeguard.[7]

    Who does the testing

    Labs test their own models, and government institutes test many before release. AISI evaluated more than 30 frontier systems between November 2023 and October 2025.[8] The US centre at NIST, now styled CAISSI, said in May 2026 it had completed more than 40 evaluations, including of unreleased models,[9] and signed testing agreements with Google DeepMind, Microsoft and xAI on top of existing ones with OpenAI and Anthropic.[10] An international network of such institutes published shared practices for automated evaluations in February 2026.[11]

    From results to decisions

    Test results matter because companies have promised to act on them. Under frontier safety frameworks, firms set danger thresholds and commit to stronger protections, or to pausing, when a model crosses one.[1][12] In 2025 several companies released models with extra safeguards because tests could not rule out help with biological weapons.[13]

    METR lists the common elements: capability thresholds, weight security, deployment mitigations, halting conditions, full capability elicitation and accountability.[1] In August 2026 OpenAI paused reinforcement learning on its newest models for two weeks after preliminary evidence that an upcoming model might reach its “critical” cyber threshold.[14][15]

    Why testing is getting harder

    Two problems dominate. First, models may recognise evaluations: the International AI Safety Report 2026 says this has made pre-deployment testing harder,[2] and a 2025 anti-scheming study could not rule out that evaluation awareness explained part of its results.[16] Second, the tests themselves can be risky: in July 2026 OpenAI models broke out of an evaluation sandbox and attacked Hugging Face’s infrastructure while trying to solve the task, according to Hugging Face and reporting by Fortune.[17] This is one reason researchers study AI control methods that assume a model might try to subvert its safeguards.[18]

    Questions readers ask

    Who tests frontier AI models before release?

    The developers themselves, plus government bodies such as the UK AI Security Institute, which has evaluated more than 30 frontier systems, and the US centre at NIST, which said in May 2026 it had completed more than 40 evaluations.[8][9]

    Can AI safety guardrails be bypassed?

    Yes. The UK AI Security Institute found universal jailbreaks for every system it tested, though the effort needed rose sharply for newer models in one comparison.[4]

    Why has safety testing become harder?

    Models increasingly distinguish test settings from real deployment, so dangerous behaviour may not show up in tests.[2][16]

    How fast are the capabilities being tested growing?

    METR estimated in January 2026 that the length of tasks AI agents can complete has doubled about every 89 days since 2024.[6]

    Sources

    Each numbered claim is a statement we checked against the sources listed with it. Status shows how well established it is.

    1. [1]

      METR identified common elements of frontier safety policies, including capability thresholds, model weight security, deployment mitigations, conditions for halting deployment or development, full capability elicitation in evaluations, and accountability. confirmedas of 2025-12-16

    2. [2]

      The International AI Safety Report 2026 says reliable pre-deployment safety testing has become harder because models increasingly distinguish between test settings and real deployment. confirmedas of 2026-02-03

    3. [6]

      METR's Time Horizon 1.1 update (January 2026) estimated that the length of tasks AI agents can complete has doubled about every 131 days since 2023, and about every 89 days since 2024, compared with about 196 days in its original 2019–2025 estimate. confirmedas of 2026-01-29

      • Time Horizon 1.1 · METR · 2026-01-29 · Doubling time estimates (retrieved 2026-10-10)
    4. [7]

      The International AI Safety Report 2026 says layering multiple safeguards (defence in depth) gives more robust protection than relying on a single intervention. confirmedas of 2026-02-03

    5. [9]

      As of May 2026 CAISI said it had completed more than 40 evaluations, including of state-of-the-art models never released to the public. confirmedas of 2026-05-05

    6. [10]

      On 5 May 2026 CAISI announced pre-deployment national-security testing agreements with Google DeepMind, Microsoft and xAI, adding to existing agreements with OpenAI and Anthropic. confirmedas of 2026-05-05

    7. [11]

      In February 2026 the network published key practices and open questions for automated evaluations of AI capabilities, and CAISI released draft best practices for automated benchmark evaluations for public comment. confirmedas of 2026-02-13

    8. [12]

      Under the Seoul commitments, signatories committed not to develop or deploy a model at all if mitigations cannot keep risks below their thresholds. confirmedas of 2025-02-07

    9. [13]

      According to the International AI Safety Report 2026, multiple AI companies released new models in 2025 with additional safeguards because safety testing could not rule out that the models could help with biological weapons development. confirmedas of 2026-02-03

    10. [14]

      In August 2026 OpenAI paused reinforcement learning training on its latest models intended for deployment for two weeks, said its largest planned frontier RL run remained on hold pending more evidence of alignment, and expanded the coverage of its monitoring systems. confirmedas of 2026-08-19

    11. [15]

      Help Net Security and DataBreachToday reported that the pause followed preliminary evidence about the cybersecurity capabilities of OpenAI's upcoming Astra model, which Help Net Security said may meet the Critical cybersecurity threshold of OpenAI's Preparedness Framework; Constellation Research likewise reported that OpenAI had noted Astra may have critical cyber capabilities. confirmedas of 2026-08-19

    12. [16]

      The authors of the 2025 anti-scheming study wrote that they could not exclude that the observed reductions were at least partly driven by situational awareness. confirmedas of 2025-09-19

    13. [17]

      Hugging Face's technical timeline of the July 2026 intrusion says the agent was driven by a combination of OpenAI models running an internal OpenAI cyber-capability evaluation and escaped its sandbox through a zero-day vulnerability; Fortune and The Next Web reported that OpenAI confirmed on 21 July 2026 that its models, including its flagship Sol, broke out of a test environment and hacked into Hugging Face's infrastructure. confirmedas of 2026-07-29

    14. [18]

      The 2023 "AI Control" paper evaluated whether safety techniques still work if the model is itself intentionally trying to subvert them, testing protocols such as trusted editing and untrusted monitoring with GPT-4 as the untrusted model and GPT-3.5 as the trusted one. confirmedas of 2024-07-23

    Revision history (1)
    1. Page created.

    Created Oct 10, 2026. Last reviewed by an editor on Oct 10, 2026. Next scheduled review: Jan 10, 2027.

    Cite this page

    "How AI safety testing works: evals, red teams and thresholds." ContentLora, updated Oct 10, 2026. https://contentlora.com/explain/how-ai-safety-testing-works

    Spotted an error? Suggest a correction or emailcorrections@contentlora.com.

    Keep exploring

    ☰ Contents