AI summary by TwinMind
Gradient Descent, Step-by-Step
StatQuest with Josh Starmer
23m 54s
Introduction and Overview
- Gradient Descent algorithm for parameter estimation
- Presenter: Josh Starmer, StatQuest
- Prerequisites: understanding of least squares and linear regression
- Applications: linear regression (intercept/slope), logistic regression (squiggle), t-SNE (clusters)
- Universal optimization strategy across statistics, ML, data science
Dataset and Problem Setup
- Simple dataset: x-axis = weight, y-axis = height
- Goal: fit line to predict height from weight (e.g., weight 1.5 → height 1.9)
- Two-phase approach:
- Phase 1: Find intercept only (slope fixed at 0.64 from least squares)
- Phase 2: Find both intercept and slope simultaneously
Phase 1: Finding Intercept Only
Initial Setup
- Fixed slope = 0.64 (least squares estimate)
- Random initial intercept = 0
- Loss function: sum of squared residuals
- Example calculation with 3 data points:
- Point 1: weight 0.5, height 1.4 → predicted 0.32 → residual 1.1
- Point 2: weight 2.3, height 1.9 → residual 0.4
- Point 3: weight 2.9, height 3.2 → residual 1.3
- Sum of squared residuals = 3.1
Gradient Descent vs Brute Force
- Brute force: test many intercept values, find minimum
- Gradient Descent: efficient approach using derivatives
- Takes big steps when far from optimum
- Takes small steps when close to optimum
Derivative Calculation
- Derived sum of squared residuals equation for all data points
- Applied chain rule to get derivative with respect to intercept
- Final derivative: -2(1.4-0.5×intercept-0.64×0.5) - 2(1.9-intercept-0.64×2.3) - 2(3.2-intercept-0.64×2.9)
Step-by-Step Optimization
- Initial intercept = 0 → derivative = -5.7
- Learning rate = 0.1
- Step size = slope × learning rate = -5.7 × 0.1 = -0.57
- New intercept = old intercept - step size = 0 - (-0.57) = 0.57
- Subsequent steps:
- Step 2: intercept = 0.8, step size = -0.23
- Step 3: intercept = 0.89, step size = -0.09
- Steps 4-6: intercept progresses to 0.95
- Final result: intercept = 0.95 (matches least squares)
Stopping Criteria
- Stop when step size < 0.001 (or smaller threshold)
- Maximum step limit: 1000+ steps as backup
- Example: slope = 0.009, learning rate = 0.1 → step size = 0.0009 < 0.001 → stop
Phase 2: Finding Both Intercept and Slope
3D Loss Function
- Loss function surface with intercept, slope, and sum of squared residuals axes
- Need partial derivatives for both parameters (gradient)
Gradient Calculation
- Partial derivative with respect to intercept: same as Phase 1
- Partial derivative with respect to slope:
- Applied chain rule, treating intercept as constant
- Includes weight values (0.5, 2.3, 2.9) in derivative terms
- Two derivatives together = gradient
Two-Parameter Optimization
- Initial values: intercept = 0, slope = 1
- Learning rate = 0.01 (smaller than Phase 1 due to sensitivity)
- Calculate two step sizes simultaneously
- Update both parameters each iteration
- Final result: intercept = 0.95, slope = 0.64 (matches least squares)
Learning Rate Sensitivity
- Larger learning rates can prevent convergence
- Practice: start large, decrease with each step (automatic determination)
- Critical parameter requiring careful tuning
General Gradient Descent Algorithm
Universal Steps
- Take derivative of loss function for each parameter (compute gradient)
- Pick random initial parameter values
- Plug parameter values into gradient
- Calculate step sizes (gradient × learning rate)
- Calculate new parameters (old - step sizes)
- Repeat steps 3-5 until convergence
Scalability to Multiple Parameters
- Method extends to any number of parameters
- Just compute more partial derivatives
- Same algorithm structure
Loss Functions and Applications
- Sum of squared residuals: one type of loss function
- Many other loss functions for different data types
- Gradient Descent works identically regardless of loss function choice
Stochastic Gradient Descent
- Standard GD: uses full dataset each step
- Problem: computationally expensive with millions of data points
- SGD solution: use randomly selected subset of data each step
- Reduces computation time for derivative calculations
- Same fundamental algorithm, different data sampling strategy
