AI summary by TwinMind

Backpropagation Details Pt. 1: Optimizing 3 parameters simultaneously.

StatQuest with Josh Starmer

18m 31s

Introduction & Prerequisites

  • StatQuest episode on backpropagation details part 1
  • Assumes prior viewing of "Neural Networks Part 2: Back Propagation Main Ideas"
  • Builds on simple neural network example with drug dosage effectiveness data

Previous Context Review

  • Simple dataset: drug dosages vs virus effectiveness
  • Neural network with optimal parameters except last bias term b₃
  • Demonstrated backpropagation main ideas by optimizing only b₃
  • Used chain rule to calculate derivative of sum of squared residuals
  • Applied gradient descent with b₃ initialized to 0

Current Objective

  • Extend optimization to multiple parameters: w₃, w₄, and b₃
  • Assume other weights/biases already optimized
  • Goal: learn chain rule application to multiple parameters
  • Introduce fancy mathematical notation
  • Next part will optimize all 7 parameters simultaneously

Parameter Initialization

  • w₃ and w₄: random values from standard normal distribution
    • w₃ = 0.36 (random initial value)
    • w₄ = 0.63 (random initial value)
  • b₃ = 0 (bias terms typically start at zero)

Neural Network Processing Flow

  • Top node connection: multiplies dosage coordinates by w₃ = 0.36
  • Bottom node connection: multiplies dosage coordinates by w₄ = 0.63
  • Blue curve: top node output × w₃
  • Orange curve: bottom node output × w₄
  • Green squiggle: sum of blue + orange curves + b₃
  • Initial sum of squared residuals = 1.4

Fancy Notation Introduction

  • i = index for dataset (i=1,2,3 for dosages 0, 0.5, 1)
  • input_i = dosage_i
  • x₁,ᵢ = x-axis coordinate for top activation function
    • x₁,ᵢ = input_i × w₁ + b₁ = input_i × 3.34 + (-1.43)
  • x₂,ᵢ = x-axis coordinate for bottom activation function
    • x₂,ᵢ = input_i × w₂ + b₂ = input_i × (-3.53) + 0.57
  • Example for i=3 (dosage=1): x₁,₃ = 1.91, x₂,₃ = -2.96
  • y₁,ᵢ = softplus(x₁,ᵢ) = y-axis coordinates for top activation
  • y₂,ᵢ = softplus(x₂,ᵢ) = y-axis coordinates for bottom activation

Chain Rule Derivatives

  • Predicted values equation: y₁,ᵢ × w₃ + y₂,ᵢ × w₄ + b₃
  • Sum of squared residuals linked to w₃, w₄ via predicted values
  • Derivative w.r.t. w₃: (∂SSR/∂predicted) × (∂predicted/∂w₃)
  • Derivative w.r.t. w₄: (∂SSR/∂predicted) × (∂predicted/∂w₄)
  • ∂SSR/∂predicted = -2(observed_i - predicted_i) (same for all parameters)

Specific Derivative Calculations

  • ∂predicted/∂w₃ = y₁,ᵢ (other terms = 0)
  • ∂predicted/∂w₄ = y₂,ᵢ (other terms = 0)
  • Final derivatives:
    • ∂SSR/∂w₃ = Σ[-2(observed_i - predicted_i) × y₁,ᵢ]
    • ∂SSR/∂w₄ = Σ[-2(observed_i - predicted_i) × y₂,ᵢ]

Gradient Descent Implementation

  • Calculate derivative for w₃: expand summation, plug in observed/predicted values, plug in y₁,ᵢ coordinates
  • Result: ∂SSR/∂w₃ = 2.58
  • Similarly calculate derivatives for w₄ and b₃
  • Update parameters using gradient descent formula
  • Repeat until convergence criteria met (predictions stop improving, max steps reached, etc.)

Results Demonstration

  • Animation shows 175 gradient descent steps
  • Gray dots = training data
  • Orange/blue curves = weighted activation outputs
  • Green squiggle = final predictions (sum + b₃)
  • Demonstrates successful parameter optimization

Next Steps

  • Part 2 will optimize all neural network parameters simultaneously
  • "Go totally bonkers with chain rule"
  • Optimize all 7 parameters at once

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome