AI summary by TwinMind

Neural Networks Pt. 2: Backpropagation Main Ideas

StatQuest with Josh Starmer

17m 33s

Prerequisites and Introduction

  • StatQuest tutorial by Josh Starmer on neural networks back propagation
  • Assumes familiarity with: neural networks, chain rule, gradient descent
  • References Part 1: "Inside the Black Box" - neural network fitting green squiggle to drug dosage effectiveness data
  • Low/high dosages ineffective, medium dosage effective against virus

Neural Network Parameter Naming Convention

  • Weights labeled: w₁, w₂, w₃, w₄
  • Biases labeled: b₁, b₂, b₃
  • Color coding system: optimized parameters = green, unoptimized = red
  • Back propagation works backwards from last parameter

Simplified Example Setup

  • Focus on optimizing final bias b₃ only
  • Assume all other parameters already optimal
  • Dosage scale: 0 (low) to 1 (high) for simplified math
  • Neural network has hidden layer with 2 nodes, output layer with 1 node

Hidden Layer Processing

  • Top node: dosages 0-1 → x-axis coordinates → soft plus activation function → multiply by -1.22 → blue curve
  • Bottom node: dosages 0-1 → x-axis coordinates → soft plus activation function → multiply by -2.3 → orange curve
  • Sum blue + orange curves = green squiggle

Initial Bias Setting and Residual Calculation

  • b₃ initialized to 0 (common practice)
  • Adding 0 to green squiggle leaves it unchanged
  • Calculate sum of squared residuals:
    • Residual 1: observed (0) - predicted (-2.6) = 2.6
    • Residual 2: observed (1) - predicted (-1.61) = 2.61
    • Residual 3: observed (0) - predicted (-2.61) = 2.61
    • Sum of squared residuals = 20.4 when b₃ = 0

Testing Different Bias Values

  • b₃ = 1: sum of squared residuals = 7.8
  • b₃ = 2: sum of squared residuals = 1.11
  • b₃ = 3: sum of squared residuals = 0.46
  • Forms pink curve showing relationship between b₃ and error

Chain Rule Application for Derivative

  • Need derivative of sum of squared residuals with respect to b₃
  • Sum notation: Σᵢ₌₁ⁿ (observed_i - predicted_i)²
  • Chain rule: d(SSR)/db₃ = d(SSR)/d(predicted) × d(predicted)/db₃
  • Predicted values = blue curve + orange curve + b₃

Derivative Calculations

  • d(SSR)/d(predicted) = -2Σ(observed_i - predicted_i)
  • d(predicted)/db₃ = 1 (since blue/orange curves independent of b₃)
  • Final derivative: d(SSR)/db₃ = -2Σ(observed_i - predicted_i)

Gradient Descent Implementation

  • Learning rate = 0.1
  • Initial calculation: derivative = -15.7 when b₃ = 0
  • Step size = -1.57
  • New b₃ = 0 + 1.57 = 1.57
  • Iteration 2: derivative = -6.26, new b₃ = 2.19
  • Continue until step size ≈ 0
  • Optimal b₃ = 2.61

Main Ideas Summary

  • Use chain rule to calculate derivative of loss function with respect to unknown parameter
  • Initialize unknown parameter with starting value
  • Apply gradient descent to optimize parameter
  • Process works backwards through network (hence "back propagation")

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome