AI summary by TwinMind
Reinforcement Learning with Neural Networks: Mathematical Details
StatQuest with Josh Starmer
25m 1s
Introduction and Prerequisites
- Josh Starmer presents mathematical details of reinforcement learning with neural networks
- Focus on policy gradients method for training neural networks
- Assumes familiarity with gradient descent and essential RL concepts
- References previous StatQuest episodes on RL fundamentals
Neural Network Setup and Example Context
- Example scenario: choosing between Squatch's Fry Shack and Norm's Fry Hut based on hunger level
- Neural network decides where to go based on hunger input (0.0 to 1.0)
- Cannot use standard backpropagation due to unknown reward outcomes
- Uses reinforcement learning with policy gradients instead
First Training Iteration - Going to Squatch's
- Input: hunger level 0.0 (not hungry)
- Neural network output: P(Norm) = 0.5, P(Squatch) = 0.5
- Random selection: 0.2 → go to Squatch's
- Guess: going to Squatch's is correct decision
- Cross entropy quantifies difference between ideal (1.0) and actual (0.5) probabilities
Mathematical Derivation - Cross Entropy Derivative
- Goal: optimize bias parameter using policy gradients
- Cross entropy for Squatch: C_Squatch = -log(P_Squatch) = -log(1 - P_Norm)
- Chain rule application: ∂C_Squatch/∂bias = (∂C_Squatch/∂P_Norm) × (∂P_Norm/∂x) × (∂x/∂bias)
First Term: ∂C_Squatch/∂P_Norm
- C_Squatch = -log(1 - P_Norm)
- Using chain rule with "stuff" = 1 - P_Norm
- Result: ∂C_Squatch/∂P_Norm = 1/(1 - P_Norm)
Second Term: ∂P_Norm/∂x (Sigmoid Derivative)
- P_Norm = sigmoid(x) = 1/(1 + e^(-x))
- Standard sigmoid derivative: sigmoid(x) × (1 - sigmoid(x))
- Simplified: P_Norm × (1 - P_Norm)
Third Term: ∂x/∂bias
- x = weight × hunger + bias
- ∂x/∂bias = 1
Complete Derivative
- Final form: [1/(1 - P_Norm)] × [P_Norm × (1 - P_Norm)] × 1 = P_Norm
- With P_Norm = 0.5: derivative = 0.5
Reward Determination and Policy Update
- Order fries from Squatch → receive small portion
- Small portion appropriate for hunger level 0.0 → correct guess
- Reward = +1.0 (correct decision)
- Updated derivative: 0.5 × 1.0 = 0.5 (same direction)
- Learning rate = 1.0, step size = 0.5
- New bias: 0.0 - 0.5 = -0.5
Second Training Iteration - Going to Norm's
- Updated neural network with bias = -0.5
- Input: hunger level 0.0
- New probabilities: P(Norm) = 0.4, P(Squatch) = 0.6
- Random selection: 0.9 → go to Norm's
- Guess: going to Norm's when not hungry is correct
Mathematical Derivation - Norm's Cross Entropy
- Cross entropy for Norm: C_Norm = -log(P_Norm)
- Derivative calculation similar to previous
- Key difference: ∂C_Norm/∂P_Norm = -1/P_Norm
- Complete derivative: (-1/P_Norm) × [P_Norm × (1 - P_Norm)] × 1 = -(1 - P_Norm)
- With P_Norm = 0.4: derivative = -0.6
Reward and Correction
- Order fries from Norm → receive huge portion
- Large portion inappropriate for hunger level 0.0 → wrong guess
- Reward = -1.0 (incorrect decision)
- Updated derivative: -0.6 × (-1.0) = 0.6 (flipped direction)
- Step size = 0.6
- New bias: -0.5 - 0.6 = -1.1
Training Completion and Results
- Continue training with various input values (0 to 1)
- Use same derivative equations with different rewards
- Final bias value stabilizes around -10
- Training complete when bias stops changing significantly
- Network learns appropriate decision-making based on hunger level
Key Reinforcement Learning Principles
- Multiply derivative by reward to correct direction
- Positive reward (+1): maintain original derivative direction
- Negative reward (-1): flip derivative direction to correct mistakes
- Policy gradients method successfully trains neural network without knowing outcomes in advance
