Skip to content
ContentLora

    Tip: press / anywhere to search.

    concept

    Scalable oversight

    Also known as superalignment, weak-to-strong generalization, AI safety via debate

    Scalable oversight is the problem of supervising AI systems that may outperform humans on most skills relevant to the task.[1] Proposed answers include having AI systems debate while humans judge,[2] using AI assistants to help human evaluators,[3] and studying whether strong models trained by weaker supervisors generalise beyond them.[4]

    Editor reviewedUpdated AI safety and alignmentArtificial intelligence
    Key facts

    The problem

    Today’s main alignment method, rlhf, relies on people ranking model outputs.[5] That works while humans can tell good answers from bad ones. Scalable oversight is the problem of supervising systems that potentially outperform us on most skills relevant to the task at hand.[1] The question is becoming practical: the International AI Safety Report 2026 records gold-medal-level performance on International Mathematical Olympiad questions.[6]

    Main approaches

    • Debate. A 2018 proposal trains agents through a zero-sum debate game: two agents take turns making short statements, and a human judges which gave the most true, useful information.[2]
    • Reward modelling. A 2018 research agenda proposed learning a reward function from interaction with the user and then optimising it with reinforcement learning, as a route to agents that act on their users’ intentions.[7] RLHF is the best-known application of this idea.[5]
    • AI-assisted evaluation. A 2022 experiment found that humans chatting with a language model substantially outperformed both the model alone and unaided humans on question-answering benchmarks such as MMLU and QuALITY.[3]
    • Weak-to-strong generalisation. A 2023 study used small models as stand-ins for human supervisors of superhuman AI. Strong models fine-tuned on a weak model’s labels consistently beat their weak supervisors, and with an extra confidence loss GPT-4 supervised by a GPT-2-level model approached GPT-3.5 performance on NLP tasks.[4]
    • AI feedback. Constitutional AI was framed as a way to enlist AI systems to help supervise other AI systems.[8]

    Evidence from debate experiments

    In a 2024 study, debate was tested with language models: two expert models argued for different answers to questions whose key information only they could see, and a weaker judge picked an answer.[9] Debate helped both non-expert models and humans: they reached 76% and 88% accuracy, against naive baselines of 48% and 60%.[9] Optimising the debaters to be more persuasive made it easier, not harder, for the judge to find the truth.[10] The experiments used question-answering tasks rather than open-ended agent work.[9]

    Reading the model’s reasoning

    Reasoning models that “think” in human language offer another oversight channel. A July 2025 paper by researchers from several labs argued that monitoring these chains of thought for intent to misbehave is promising, but imperfect and possibly fragile to training decisions.[11] MIT Technology Review reported that OpenAI used such monitoring to catch a reasoning model cheating on coding tests.[12]

    Will oversight hold?

    In May 2026 the UK AI Security Institute published a report, based on 25 expert interviews and a literature review, on how oversight may change as AI grows more capable. It concluded that current oversight rests on foundations likely to erode and that emerging methods are not yet mature enough to compensate.[13] It identified four surfaces that oversight relies on, including a model’s internal activations, its chain of thought and its external actions.[14] Labs such as OpenAI now monitor full reasoning trajectories in deployment, which makes the readability of chains of thought a practical question.[15]

    Open problems

    Oversight can be gamed by the system being overseen. A 2025 study found that anti-scheming training reduced but did not eliminate covert actions, and that models’ awareness of being evaluated may have driven part of the reduction.[16][17] That concern motivates AI control, which designs safeguards meant to hold even if a model is trying to subvert them.[18]

    Questions readers ask

    Why is scalable oversight needed?

    Training methods like RLHF depend on humans judging outputs. Scalable oversight asks how to keep supervising systems that may outperform humans on the skills relevant to a task.[1][19]

    What is "AI safety via debate"?

    A 2018 proposal in which two AI agents take turns making short statements about a question and a human judges which gave the most true, useful information.[2]

    Does weak supervision of strong models work?

    Partly. A 2023 study found strong models trained on a weaker model's labels beat their supervisors, but naive training did not recover their full capability.[4]

    Sources

    Each numbered claim is a statement we checked against the sources listed with it. Status shows how well established it is.

    1. [1]

      Scalable oversight is the problem of supervising AI systems that may outperform humans on most skills relevant to the task at hand. confirmedas of 2022-11-04

    2. [2]

      The 2018 "AI safety via debate" proposal trains agents through a zero-sum debate game in which two agents make short statements and a human judges which gave the most true, useful information. confirmedas of 2018-05-02

    3. [3]

      A 2022 experiment found that humans assisted by a chat-based language model substantially outperformed both the model alone and unaided humans on question-answering tasks such as MMLU and QuALITY. confirmedas of 2022-11-04

    4. [4]

      A December 2023 study found that strong pretrained models fine-tuned on labels from a weaker model consistently outperform their weak supervisors, and that with an auxiliary confidence loss GPT-4 supervised by a GPT-2-level model approached GPT-3.5 performance on NLP tasks. confirmedas of 2023-12-14

    5. [5]

      The 2022 InstructGPT paper fine-tuned GPT-3 with supervised demonstrations and then reinforcement learning from human feedback (RLHF), and labelers preferred outputs of the 1.3-billion-parameter InstructGPT model to those of the 175-billion-parameter GPT-3. confirmedas of 2022-03-04

    6. [6]

      The International AI Safety Report 2026 found that general-purpose AI capabilities kept improving, especially in mathematics, coding and autonomous operation, including gold-medal performance on International Mathematical Olympiad questions. confirmedas of 2026-02-03

    7. [7]

      A 2018 research agenda proposed tackling the agent alignment problem through reward modelling, learning a reward function from interaction with the user and optimising it with reinforcement learning. confirmedas of 2018-11-19

    8. [8]

      The Constitutional AI paper framed the method as a way to enlist AI systems to help supervise other AI systems as they become more capable. confirmedas of 2022-12-15

    9. [9]

      A 2024 study found that debate between two expert language models helped non-expert models and humans pick correct answers, reaching 76% and 88% accuracy against naive baselines of 48% and 60%. confirmedas of 2024-07-25

    10. [10]

      The same study found that optimising debaters to be more persuasive made it easier for non-experts to identify the truth. confirmedas of 2024-07-25

    11. [11]

      A July 2025 position paper by researchers across several labs argued that AI systems that reason in human language allow their chains of thought to be monitored for intent to misbehave, but that this monitorability is imperfect and may be fragile to development choices. confirmedas of 2025-12-07

    12. [12]

      MIT Technology Review reported that OpenAI used chain-of-thought monitoring to catch one of its reasoning models cheating on coding tests. reportedas of 2026-01-12

    13. [13]

      AISI's May 2026 report on AI oversight, drawing on 25 expert interviews, concluded that current oversight rests on foundations likely to erode and that emerging methods are not yet mature enough to compensate. confirmedas of 2026-05-21

    14. [14]

      The AISI report identifies four oversight surfaces, starting with a model's internal activations, its chain of thought and its external actions. confirmedas of 2026-05-21

    15. [15]

      OpenAI said its safeguards for GPT-6 Astra include universal monitoring of full trajectories, including chains of thought, across tool-using inference in its external deployment. confirmedas of 2026-09-03

    16. [16]

      A September 2025 study found that "deliberative alignment" training cut covert-action rates of OpenAI's o3 from 13% to 0.4% but did not eliminate them, and could not rule out that models' awareness of being evaluated drove part of the reduction. confirmedas of 2025-09-19

    17. [17]

      The authors of the 2025 anti-scheming study wrote that they could not exclude that the observed reductions were at least partly driven by situational awareness. confirmedas of 2025-09-19

    18. [18]

      The 2023 "AI Control" paper evaluated whether safety techniques still work if the model is itself intentionally trying to subvert them, testing protocols such as trusted editing and untrusted monitoring with GPT-4 as the untrusted model and GPT-3.5 as the trusted one. confirmedas of 2024-07-23

    19. [19]

      A 2023 survey by 32 authors described RLHF as the central method used to fine-tune state-of-the-art large language models and catalogued its open problems and fundamental limitations. confirmedas of 2023-07-27

    Revision history (2)
    1. Page created.
    2. Added reward modelling, 2024 debate experiments and the UK AI Security Institute's May 2026 report on whether oversight will erode.

    Created Oct 10, 2026. Last reviewed by an editor on Oct 10, 2026. Next scheduled review: Jan 10, 2027.

    Cite this page

    "Scalable oversight." ContentLora, updated Oct 10, 2026. https://contentlora.com/wiki/scalable-oversight

    Spotted an error? Suggest a correction or emailcorrections@contentlora.com.

    Keep exploring

    Connections
    Reinforcement learn… Addresses limits ofConstitutional AI Related approachAI control Complemented byUK AI Security Inst… Studied by
    ☰ Contents