AI summary by TwinMind

Reinforcement Learning with Neural Networks: Mathematical Details

StatQuest with Josh Starmer

25m 1s

Introduction and Prerequisites

  • Josh Starmer presents mathematical details of reinforcement learning with neural networks
  • Focus on policy gradients method for training neural networks
  • Assumes familiarity with gradient descent and essential RL concepts
  • References previous StatQuest episodes on RL fundamentals

Neural Network Setup and Example Context

  • Example scenario: choosing between Squatch's Fry Shack and Norm's Fry Hut based on hunger level
  • Neural network decides where to go based on hunger input (0.0 to 1.0)
  • Cannot use standard backpropagation due to unknown reward outcomes
  • Uses reinforcement learning with policy gradients instead

First Training Iteration - Going to Squatch's

  • Input: hunger level 0.0 (not hungry)
  • Neural network output: P(Norm) = 0.5, P(Squatch) = 0.5
  • Random selection: 0.2 → go to Squatch's
  • Guess: going to Squatch's is correct decision
  • Cross entropy quantifies difference between ideal (1.0) and actual (0.5) probabilities

Mathematical Derivation - Cross Entropy Derivative

  • Goal: optimize bias parameter using policy gradients
  • Cross entropy for Squatch: C_Squatch = -log(P_Squatch) = -log(1 - P_Norm)
  • Chain rule application: ∂C_Squatch/∂bias = (∂C_Squatch/∂P_Norm) × (∂P_Norm/∂x) × (∂x/∂bias)

First Term: ∂C_Squatch/∂P_Norm

  • C_Squatch = -log(1 - P_Norm)
  • Using chain rule with "stuff" = 1 - P_Norm
  • Result: ∂C_Squatch/∂P_Norm = 1/(1 - P_Norm)

Second Term: ∂P_Norm/∂x (Sigmoid Derivative)

  • P_Norm = sigmoid(x) = 1/(1 + e^(-x))
  • Standard sigmoid derivative: sigmoid(x) × (1 - sigmoid(x))
  • Simplified: P_Norm × (1 - P_Norm)

Third Term: ∂x/∂bias

  • x = weight × hunger + bias
  • ∂x/∂bias = 1

Complete Derivative

  • Final form: [1/(1 - P_Norm)] × [P_Norm × (1 - P_Norm)] × 1 = P_Norm
  • With P_Norm = 0.5: derivative = 0.5

Reward Determination and Policy Update

  • Order fries from Squatch → receive small portion
  • Small portion appropriate for hunger level 0.0 → correct guess
  • Reward = +1.0 (correct decision)
  • Updated derivative: 0.5 × 1.0 = 0.5 (same direction)
  • Learning rate = 1.0, step size = 0.5
  • New bias: 0.0 - 0.5 = -0.5

Second Training Iteration - Going to Norm's

  • Updated neural network with bias = -0.5
  • Input: hunger level 0.0
  • New probabilities: P(Norm) = 0.4, P(Squatch) = 0.6
  • Random selection: 0.9 → go to Norm's
  • Guess: going to Norm's when not hungry is correct

Mathematical Derivation - Norm's Cross Entropy

  • Cross entropy for Norm: C_Norm = -log(P_Norm)
  • Derivative calculation similar to previous
  • Key difference: ∂C_Norm/∂P_Norm = -1/P_Norm
  • Complete derivative: (-1/P_Norm) × [P_Norm × (1 - P_Norm)] × 1 = -(1 - P_Norm)
  • With P_Norm = 0.4: derivative = -0.6

Reward and Correction

  • Order fries from Norm → receive huge portion
  • Large portion inappropriate for hunger level 0.0 → wrong guess
  • Reward = -1.0 (incorrect decision)
  • Updated derivative: -0.6 × (-1.0) = 0.6 (flipped direction)
  • Step size = 0.6
  • New bias: -0.5 - 0.6 = -1.1

Training Completion and Results

  • Continue training with various input values (0 to 1)
  • Use same derivative equations with different rewards
  • Final bias value stabilizes around -10
  • Training complete when bias stops changing significantly
  • Network learns appropriate decision-making based on hunger level

Key Reinforcement Learning Principles

  • Multiply derivative by reward to correct direction
  • Positive reward (+1): maintain original derivative direction
  • Negative reward (-1): flip derivative direction to correct mistakes
  • Policy gradients method successfully trains neural network without knowing outcomes in advance

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome