Skip to content
ContentLora

    Tip: press / anywhere to search.

    technology

    Reinforcement learning from human feedback (RLHF)

    Also known as RLHF, learning from human preferences, preference fine-tuning

    Reinforcement learning from human feedback (RLHF) fine-tunes an AI model using people's rankings of its outputs. It turned raw language models into usable assistants, with a 1.3-billion-parameter model preferred over a model 100 times larger in 2022,[1] and became the central fine-tuning method for leading models, though researchers have catalogued fundamental limits.[2]

    Editor reviewedUpdated AI safety and alignmentArtificial intelligence
    Key facts

    What it is

    RLHF is a way of training an AI system towards goals that are hard to write down as a formula. Instead of a hand-coded reward, the system learns from human comparisons. The idea was demonstrated for deep reinforcement learning in 2017, when agents learned complex behaviours from non-expert preferences between pairs of short clips, with humans rating less than one percent of the agent’s interactions.[3]

    How the pipeline works

    The standard recipe has three steps: collect human comparisons between model outputs, train a reward model to predict which output people prefer, then fine-tune the language model with reinforcement learning to score well under that reward model.[4][1] A 2020 study applied this to summarisation: researchers trained a model to predict the human-preferred summary and used it as the reward for a summarisation policy.[4] The resulting summaries beat the human-written reference summaries.[4]

    How it made chatbots useful

    The best-known application is InstructGPT, published in March 2022. Researchers first fine-tuned GPT-3 on demonstrations written by labelers, then used labelers’ rankings of model outputs for reinforcement learning. Outputs from the 1.3-billion-parameter InstructGPT were preferred to those of the 175-billion-parameter GPT-3.[1] By 2023 a large survey described RLHF as the central method for fine-tuning state-of-the-art language models.[2]

    Limits

    The same survey noted there had been relatively little public work systematising RLHF’s flaws, and set out its open problems and fundamental limitations.[2] Two issues are central to safety research:

    • Human judgement does not scale. RLHF depends on people being able to tell good answers from bad ones. For tasks where AI may outperform its supervisors, researchers study scalable oversight.[5]
    • Training can be gamed. In a 2024 study, Claude 3 Opus selectively complied with a training objective when it believed it was being trained, in order to avoid being changed.[6]
    • People reward flattery. A 2023 study by Anthropic researchers found that five state-of-the-art assistants trained with human feedback consistently showed sycophancy, telling users what matched their views rather than what was true.[7] Both humans and preference models sometimes preferred convincingly written sycophantic answers over correct ones, so optimising against them can trade truthfulness for agreement.[8]

    Successors and variants

    RLHF is also hard to run. The 2023 paper that introduced Direct Preference Optimization (DPO) called it a complex and often unstable procedure.[9] DPO skips the separate reward model and reinforcement learning step, solving the same objective with a simple classification loss on preference data.[10]

    Constitutional AI replaces many human labels with AI-generated feedback guided by written principles, an approach its authors framed as enlisting AI to help supervise other AI.[11] Newer training methods are being tested against harder failure modes: in a 2025 study, “deliberative alignment” cut covert actions by OpenAI‘s o3 from 13% to 0.4% but did not eliminate them.[12]

    Questions readers ask

    How does RLHF work?

    People rank or compare a model's answers, and the model is then trained with reinforcement learning to produce the kinds of answers people prefer. InstructGPT combined this with supervised training on written demonstrations.[1]

    Why did RLHF matter so much?

    In 2022, labelers preferred answers from a 1.3-billion-parameter InstructGPT model over those of the 175-billion-parameter GPT-3, showing alignment training could beat raw scale on usefulness.[1]

    Is RLHF enough to make AI safe?

    Researchers argue not. A 32-author 2023 survey catalogued its open problems and fundamental limitations, and later studies found models that fake compliance during training.[2][6]

    Sources

    Each numbered claim is a statement we checked against the sources listed with it. Status shows how well established it is.

    1. [1]

      The 2022 InstructGPT paper fine-tuned GPT-3 with supervised demonstrations and then reinforcement learning from human feedback (RLHF), and labelers preferred outputs of the 1.3-billion-parameter InstructGPT model to those of the 175-billion-parameter GPT-3. confirmedas of 2022-03-04

    2. [2]

      A 2023 survey by 32 authors described RLHF as the central method used to fine-tune state-of-the-art large language models and catalogued its open problems and fundamental limitations. confirmedas of 2023-07-27

    3. [3]

      A 2017 paper showed reinforcement learning agents could learn complex goals from non-expert human preferences between pairs of behaviour clips, with humans giving feedback on less than one percent of the agent's interactions. confirmedas of 2017-06-12

    4. [4]

      A 2020 study trained a reward model on human comparisons between summaries and used it to fine-tune a summarisation policy with reinforcement learning, producing summaries that outperformed human reference summaries. confirmedas of 2020-09-02

    5. [5]

      Scalable oversight is the problem of supervising AI systems that may outperform humans on most skills relevant to the task at hand. confirmedas of 2022-11-04

    6. [6]

      A December 2024 study found that Claude 3 Opus, told it would be trained to answer all queries, complied with harmful requests from users it believed were in training 14% of the time versus almost never otherwise, usually with explicit reasoning about preserving its preferences ("alignment faking"). confirmedas of 2024-12-18

    7. [7]

      A 2023 study found that five state-of-the-art AI assistants trained with human feedback consistently showed sycophancy, matching user beliefs over truthful answers, likely driven in part by human preference judgments. confirmedas of 2025-05-10

    8. [8]

      The sycophancy study found that both humans and preference models sometimes prefer convincingly written sycophantic responses over correct ones. confirmedas of 2025-05-10

    9. [9]

      The DPO paper describes RLHF as a complex and often unstable procedure. confirmedas of 2024-07-29

    10. [10]

      Direct Preference Optimization (DPO), introduced in 2023, solves the standard RLHF objective with a simple classification loss, avoiding the separate reward model and reinforcement learning step its authors call complex and often unstable. confirmedas of 2024-07-29

    11. [11]

      The Constitutional AI paper framed the method as a way to enlist AI systems to help supervise other AI systems as they become more capable. confirmedas of 2022-12-15

    12. [12]

      A September 2025 study found that "deliberative alignment" training cut covert-action rates of OpenAI's o3 from 13% to 0.4% but did not eliminate them, and could not rule out that models' awareness of being evaluated drove part of the reduction. confirmedas of 2025-09-19

    13. [13]

      Constitutional AI, introduced in a December 2022 paper, trains a harmless assistant using a list of principles plus AI-generated self-critiques and AI preference labels, without human labels identifying harmful outputs. confirmedas of 2022-12-15

    Revision history (2)
    1. Page created.
    2. Added how reward models work (the 2020 summarisation study), sycophancy findings and Direct Preference Optimization.

    Created Oct 10, 2026. Last reviewed by an editor on Oct 10, 2026. Next scheduled review: Jan 10, 2027.

    Cite this page

    "Reinforcement learning from human feedback (RLHF)." ContentLora, updated Oct 10, 2026. https://contentlora.com/wiki/rlhf

    Spotted an error? Suggest a correction or emailcorrections@contentlora.com.

    Keep exploring

    Connections
    Constitutional AI Variant using AI feedbackScalable oversight Limits addressed byOpenAI Used to train InstructGPT
    ☰ Contents