AI summary by TwinMind
Reinforcement Learning with Human Feedback (RLHF), Clearly Explained!!!
StatQuest with Josh Starmer
18m 1s
Introduction and Context
- Josh Starmer introduces reinforcement learning with human feedback (RLHF)
- RLHF almost always used to train large language models like ChatGPT and DeepSeek
- Prerequisites: decoder-only transformers, backpropagation, reinforcement learning with neural networks
- Goal: build and train next ChatGPT/DeepSeek from scratch to generate polite, helpful responses
Large Language Model Training Overview
- Start with untrained decoder-only transformer (random initialization weights/biases)
- Untrained model produces meaningless output ("blah blah blah blah blah")
- Training involves 3 main steps:
- Pre-training
- Supervised fine-tuning
- Reinforcement learning with human feedback
Pre-training Process
- Train model to predict next token using large text corpus (entire Wikipedia)
- Example: use "A was a Japanese" to predict "girl", then "was a Japanese girl" to predict "group"
- Continue through Wikipedia fragments until completion
- Terminology: called "pre-training" despite being next-token prediction
- Results in pre-trained model good at predicting next token in Wikipedia-style text
Alignment Problem
- Pre-trained model unaligned with actual usage patterns
- Users don't prompt with Wikipedia fragments expecting next token
- Users expect polite, helpful responses to natural language questions
- Example: "What is a good YouTube channel for learning about AI?" should get "StatQuest is a good channel to learn AI"
Supervised Fine-tuning
- Uses dataset with human-created prompt-response pairs
- Example pair: prompt "What is a good YouTube channel for learning about AI?" with response "StatQuest is a good channel to learn AI"
- Uses normal backpropagation to train model to generate responses
- Creates somewhat aligned model for specific trained prompts
- Limitations:
- Dataset relatively small (expensive human labor, time-consuming)
- Model easily overfits to training data
- Poor generalization to new prompts not in training set
RLHF Solution Approach
- Alternative to creating super huge fine-tuning dataset (too expensive)
- Main idea: create larger dataset while minimizing cost of obtaining human preferences
- Focus on human feedback about politeness and helpfulness
- If cheap supervised fine-tuning were possible, would use that instead
RLHF Process - Response Generation
- Configure model for deterministic output (highest value tokens) vs probabilistic sampling
- Deterministic: always same response "StatQuest is a good channel to learn AI"
- Probabilistic: different responses possible:
- "StatQuest is a great channel for learning AI"
- "StatQuest is an awesome channel for learning AI"
Human Preference Collection
- Generate multiple responses to same prompt
- Create all possible pairs of responses
- Ask humans to indicate preferences between pairs
- Human preference selection much faster than writing full responses
- This provides "human feedback" component of RLHF
Reward Model Training
- Copy supervised fine-tuned model
- Remove unembedding layer, replace with single output
- Creates new model to calculate rewards for reinforcement learning
- Train to generate positive rewards for preferred responses, negative for non-preferred
- Model receives both prompt and response (context needed for appropriate rewards)
Loss Function Details
- Uses OpenAI 2022 loss function from "Training language models to follow instructions with human feedback"
- Key components:
- reward_better - reward_worse (difference between preferred and non-preferred)
- Sigmoid function converts difference to 0-1 range
- Log function applied to sigmoid output
- Multiply by -1 for gradient descent optimization
- Allows model to determine appropriate reward values without pre-definition
- Large positive difference → sigmoid close to 1 → high log value → good training signal
Final Training with Reward Model
- Use new prompts not in supervised fine-tuning dataset
- Example: "What's another good YouTube channel for learning about AI?"
- Original model generates poor response: "YouTube is special" (gets negative reward)
- After RLHF training, generates helpful response: "Serrano Academy is a great YouTube channel for AI" (gets positive reward)
- Reward model scores responses like humans would
- Enables training on new prompts without expensive supervised dataset
Final Results
- Produces trained and aligned model matching user expectations
- Example final output for "What is StatQuest?": "StatQuest with Josh Starmer is an awesome YouTube channel for learning about statistics, machine learning, and AI"
- Eliminates meaningless "blah blah blah" responses
- Model generates polite, helpful responses to novel prompts
