AI summary by TwinMind

Reinforcement Learning with Human Feedback (RLHF), Clearly Explained!!!

StatQuest with Josh Starmer

18m 1s

Introduction and Context

  • Josh Starmer introduces reinforcement learning with human feedback (RLHF)
  • RLHF almost always used to train large language models like ChatGPT and DeepSeek
  • Prerequisites: decoder-only transformers, backpropagation, reinforcement learning with neural networks
  • Goal: build and train next ChatGPT/DeepSeek from scratch to generate polite, helpful responses

Large Language Model Training Overview

  • Start with untrained decoder-only transformer (random initialization weights/biases)
  • Untrained model produces meaningless output ("blah blah blah blah blah")
  • Training involves 3 main steps:
    • Pre-training
    • Supervised fine-tuning
    • Reinforcement learning with human feedback

Pre-training Process

  • Train model to predict next token using large text corpus (entire Wikipedia)
  • Example: use "A was a Japanese" to predict "girl", then "was a Japanese girl" to predict "group"
  • Continue through Wikipedia fragments until completion
  • Terminology: called "pre-training" despite being next-token prediction
  • Results in pre-trained model good at predicting next token in Wikipedia-style text

Alignment Problem

  • Pre-trained model unaligned with actual usage patterns
  • Users don't prompt with Wikipedia fragments expecting next token
  • Users expect polite, helpful responses to natural language questions
  • Example: "What is a good YouTube channel for learning about AI?" should get "StatQuest is a good channel to learn AI"

Supervised Fine-tuning

  • Uses dataset with human-created prompt-response pairs
  • Example pair: prompt "What is a good YouTube channel for learning about AI?" with response "StatQuest is a good channel to learn AI"
  • Uses normal backpropagation to train model to generate responses
  • Creates somewhat aligned model for specific trained prompts
  • Limitations:
    • Dataset relatively small (expensive human labor, time-consuming)
    • Model easily overfits to training data
    • Poor generalization to new prompts not in training set

RLHF Solution Approach

  • Alternative to creating super huge fine-tuning dataset (too expensive)
  • Main idea: create larger dataset while minimizing cost of obtaining human preferences
  • Focus on human feedback about politeness and helpfulness
  • If cheap supervised fine-tuning were possible, would use that instead

RLHF Process - Response Generation

  • Configure model for deterministic output (highest value tokens) vs probabilistic sampling
  • Deterministic: always same response "StatQuest is a good channel to learn AI"
  • Probabilistic: different responses possible:
    • "StatQuest is a great channel for learning AI"
    • "StatQuest is an awesome channel for learning AI"

Human Preference Collection

  • Generate multiple responses to same prompt
  • Create all possible pairs of responses
  • Ask humans to indicate preferences between pairs
  • Human preference selection much faster than writing full responses
  • This provides "human feedback" component of RLHF

Reward Model Training

  • Copy supervised fine-tuned model
  • Remove unembedding layer, replace with single output
  • Creates new model to calculate rewards for reinforcement learning
  • Train to generate positive rewards for preferred responses, negative for non-preferred
  • Model receives both prompt and response (context needed for appropriate rewards)

Loss Function Details

  • Uses OpenAI 2022 loss function from "Training language models to follow instructions with human feedback"
  • Key components:
    • reward_better - reward_worse (difference between preferred and non-preferred)
    • Sigmoid function converts difference to 0-1 range
    • Log function applied to sigmoid output
    • Multiply by -1 for gradient descent optimization
  • Allows model to determine appropriate reward values without pre-definition
  • Large positive difference → sigmoid close to 1 → high log value → good training signal

Final Training with Reward Model

  • Use new prompts not in supervised fine-tuning dataset
  • Example: "What's another good YouTube channel for learning about AI?"
  • Original model generates poor response: "YouTube is special" (gets negative reward)
  • After RLHF training, generates helpful response: "Serrano Academy is a great YouTube channel for AI" (gets positive reward)
  • Reward model scores responses like humans would
  • Enables training on new prompts without expensive supervised dataset

Final Results

  • Produces trained and aligned model matching user expectations
  • Example final output for "What is StatQuest?": "StatQuest with Josh Starmer is an awesome YouTube channel for learning about statistics, machine learning, and AI"
  • Eliminates meaningless "blah blah blah" responses
  • Model generates polite, helpful responses to novel prompts

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome