AI summary by TwinMind

Gradient Descent, Step-by-Step

StatQuest with Josh Starmer

23m 54s

Introduction and Overview

  • Gradient Descent algorithm for parameter estimation
  • Presenter: Josh Starmer, StatQuest
  • Prerequisites: understanding of least squares and linear regression
  • Applications: linear regression (intercept/slope), logistic regression (squiggle), t-SNE (clusters)
  • Universal optimization strategy across statistics, ML, data science

Dataset and Problem Setup

  • Simple dataset: x-axis = weight, y-axis = height
  • Goal: fit line to predict height from weight (e.g., weight 1.5 → height 1.9)
  • Two-phase approach:
    • Phase 1: Find intercept only (slope fixed at 0.64 from least squares)
    • Phase 2: Find both intercept and slope simultaneously

Phase 1: Finding Intercept Only

Initial Setup

  • Fixed slope = 0.64 (least squares estimate)
  • Random initial intercept = 0
  • Loss function: sum of squared residuals
  • Example calculation with 3 data points:
    • Point 1: weight 0.5, height 1.4 → predicted 0.32 → residual 1.1
    • Point 2: weight 2.3, height 1.9 → residual 0.4
    • Point 3: weight 2.9, height 3.2 → residual 1.3
    • Sum of squared residuals = 3.1

Gradient Descent vs Brute Force

  • Brute force: test many intercept values, find minimum
  • Gradient Descent: efficient approach using derivatives
    • Takes big steps when far from optimum
    • Takes small steps when close to optimum

Derivative Calculation

  • Derived sum of squared residuals equation for all data points
  • Applied chain rule to get derivative with respect to intercept
  • Final derivative: -2(1.4-0.5×intercept-0.64×0.5) - 2(1.9-intercept-0.64×2.3) - 2(3.2-intercept-0.64×2.9)

Step-by-Step Optimization

  • Initial intercept = 0 → derivative = -5.7
  • Learning rate = 0.1
  • Step size = slope × learning rate = -5.7 × 0.1 = -0.57
  • New intercept = old intercept - step size = 0 - (-0.57) = 0.57
  • Subsequent steps:
    • Step 2: intercept = 0.8, step size = -0.23
    • Step 3: intercept = 0.89, step size = -0.09
    • Steps 4-6: intercept progresses to 0.95
  • Final result: intercept = 0.95 (matches least squares)

Stopping Criteria

  • Stop when step size < 0.001 (or smaller threshold)
  • Maximum step limit: 1000+ steps as backup
  • Example: slope = 0.009, learning rate = 0.1 → step size = 0.0009 < 0.001 → stop

Phase 2: Finding Both Intercept and Slope

3D Loss Function

  • Loss function surface with intercept, slope, and sum of squared residuals axes
  • Need partial derivatives for both parameters (gradient)

Gradient Calculation

  • Partial derivative with respect to intercept: same as Phase 1
  • Partial derivative with respect to slope:
    • Applied chain rule, treating intercept as constant
    • Includes weight values (0.5, 2.3, 2.9) in derivative terms
  • Two derivatives together = gradient

Two-Parameter Optimization

  • Initial values: intercept = 0, slope = 1
  • Learning rate = 0.01 (smaller than Phase 1 due to sensitivity)
  • Calculate two step sizes simultaneously
  • Update both parameters each iteration
  • Final result: intercept = 0.95, slope = 0.64 (matches least squares)

Learning Rate Sensitivity

  • Larger learning rates can prevent convergence
  • Practice: start large, decrease with each step (automatic determination)
  • Critical parameter requiring careful tuning

General Gradient Descent Algorithm

Universal Steps

  1. Take derivative of loss function for each parameter (compute gradient)
  2. Pick random initial parameter values
  3. Plug parameter values into gradient
  4. Calculate step sizes (gradient × learning rate)
  5. Calculate new parameters (old - step sizes)
  6. Repeat steps 3-5 until convergence

Scalability to Multiple Parameters

  • Method extends to any number of parameters
  • Just compute more partial derivatives
  • Same algorithm structure

Loss Functions and Applications

  • Sum of squared residuals: one type of loss function
  • Many other loss functions for different data types
  • Gradient Descent works identically regardless of loss function choice

Stochastic Gradient Descent

  • Standard GD: uses full dataset each step
  • Problem: computationally expensive with millions of data points
  • SGD solution: use randomly selected subset of data each step
  • Reduces computation time for derivative calculations
  • Same fundamental algorithm, different data sampling strategy

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome