AI summary by TwinMind

Backpropagation Details Pt. 2: Going bonkers with The Chain Rule

StatQuest with Josh Starmer

13m 8s

Introduction & Prerequisites

  • Josh Starmer presents StatQuest on backpropagation details part 2
  • Requires viewing backpropagation details part 1 first
  • Focus: applying chain rule and gradient descent to optimize neural network weights/biases
  • Goal: fit green squiggle to data using neural network

Neural Network Structure Review

  • Derivatives from part 1 remain unchanged:
    • Bias b₃
    • Weights w₃ and w₄
  • New derivatives needed:
    • w₁, b₁, w₂, b₂
  • Network flow: input → multiply by weights → add bias → activation function → final output

Derivative of Sum of Squared Residuals w.r.t. w₁

  • Chain of connections: sum of squared residuals → predicted values → y₁,ᵢ → x₁,ᵢ → w₁
  • Chain rule application:
    • ∂(SSR)/∂w₁ = ∂(SSR)/∂(predicted) × ∂(predicted)/∂y₁ × ∂y₁/∂x₁ × ∂x₁/∂w₁
  • Individual derivatives:
    • ∂(SSR)/∂(predicted) = sum of -2(observed - predicted)
    • ∂(predicted)/∂y₁ = w₃ (first term), 0 (other terms)
    • ∂y₁/∂x₁ = e^x/(1 + e^x) (softplus activation function derivative)
    • ∂x₁/∂w₁ = input values
  • Result: "Big Fancy Equation" (BFE)

Derivative of Sum of Squared Residuals w.r.t. b₁

  • Same chain of connections as w₁ case
  • Only difference: ∂x₁/∂b₁ = 1 (instead of input values)
  • Results in another big fancy equation

Derivatives for Second Node (w₂, b₂)

  • Similar process using bottom node in hidden layer
  • Network flow: input × w₂ + b₂ → x₂,ᵢ → activation function → y₂,ᵢ → × w₄ → orange curve
  • Same chain rule methodology with parameter name changes
  • Generates derivatives for both w₂ and b₂

Gradient Descent Implementation

  • Initialize weights from standard normal distribution
  • Initialize bias terms to zero
  • Calculate all derivatives simultaneously
  • Update parameters iteratively until convergence criteria met

Example Calculation

  • Derivative calculation for w₁:
    • Expand summation
    • Plug in observed values and predicted values from green squiggle
    • Insert current w₃ value and x-axis coordinates (x₁,ᵢ)
    • Include input values
    • Result: 0.76

Training Process Visualization

  • Gray dots: training data
  • Orange and blue curves: individual neural network components
  • Green squiggle: sum of orange + blue curves + b₃
  • Animation shows gradient descent fitting green squiggle to data
  • Initial poor fit (going off screen) improves through optimization

Results

  • Successfully applied chain rule and gradient descent
  • Neural network now fits green squiggle to data
  • "Triple bam" - complete optimization achieved

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome