AI summary by TwinMind
Reinforcement Learning with Neural Networks: Essential Concepts
StatQuest with Josh Starmer
23m 59s
Introduction to Reinforcement Learning Problem
- Example scenario: choosing between Squatch's Fry Shack vs Norm's Fry Hut based on hunger level
- Traditional coin flip doesn't account for hunger level or portion sizes
- Need neural network that takes hunger as input, outputs probability of going to Norm's
- Problem: no training dataset with known ideal outputs (don't know portion sizes in advance)
Traditional Back Propagation Limitations
- Standard training requires input-output pairs with ideal values
- Calculate differences between neural network output and ideal values
- Use derivatives to determine bias adjustment direction:
- Negative derivative → increase bias (shift right)
- Positive derivative → decrease bias (shift left)
- Without ideal output values, can't calculate differences or derivatives
- Can't determine which direction to adjust parameters
Policy Gradients Solution Approach
- Use guessing instead of known ideal values
- Guess what ideal values should be, use for derivative calculations
- Core concept: "When you don't know, take a guess and see what happens"
Step-by-Step Policy Gradients Example
Initial Setup
- Hunger level input: 0.0 (not hungry)
- Neural network output: P_norm = 0.5, P_squatch = 0.5
- Random selection: 0.2 → go to Squatch's (falls in 0-0.5 range)
Making the Guess
- Guess: going to Squatch's is correct choice
- Ideal probability values based on guess: P_squatch = 1.0, P_norm = 0.0
- Calculate difference between ideal (1.0) and actual (0.5) P_squatch values
- Compute derivative with respect to bias: 0.5
Reward Determination
- Order fries at Squatch's → receive small portion
- Since hunger = 0.0 (not hungry), small portion is good
- Correct guess → reward = 1.0
- If large portion received → wrong guess → reward = -1.0
Derivative Update and Parameter Adjustment
- Updated derivative = original derivative × reward = 0.5 × 1.0 = 0.5
- Learning rate = 1.0
- Step size = 1.0 × 0.5 = 0.5
- New bias = old bias - step size = 0.0 - 0.5 = -0.5
Second Iteration Results
- With new bias (-0.5), hunger input 0.0 produces:
- P_norm = 0.4, P_squatch = 0.6
- Higher probability of returning to Squatch's (correct behavior)
- Random selection 0.9 → go to Norm's
Second Iteration Processing
- Guess: going to Norm's is correct
- Derivative calculation: -0.6
- Order fries at Norm's → receive huge portion
- Since hunger = 0.0, huge portion is bad → wrong guess
- Reward = -1.0
- Updated derivative = -0.6 × (-1.0) = 0.6
- New bias = -0.5 - 0.6 = -1.1
Training Convergence
- After many iterations with various hunger inputs (0 to 1)
- Bias converges around -10
- Final behavior:
- Hunger = 0.0 → P_norm = 0 (always go to Squatch's)
- Hunger = 1.0 → P_norm = 1 (always go to Norm's)
Key Reward System Concepts
- Rewards don't need to be ±1.0
- Larger absolute rewards create larger parameter updates
- Positive reward = correct guess, negative reward = incorrect guess
- Reward magnitude affects step size in parameter updates
Policy Gradients Summary Process
- Use neural network to make decision
- Guess that decision was optimal
- Calculate derivative based on guess
- Determine actual reward from environment
- Multiply derivative by reward for correction
- Apply gradient descent with updated derivative
- Repeat until convergence
