AI summary by TwinMind
Ridge vs Lasso Regression, Visualized!!!
StatQuest with Josh Starmer
9m 5s
Introduction & Setup
- StatQuest tutorial by Josh Starmer comparing Ridge vs Lasso regression
- Assumes prior familiarity with both regression types
- Uses simple dataset: weight and height measurements
- Demonstrates differences through visualization
Baseline Model & Residuals Calculation
- Start with horizontal line (terrible fit) to data
- Calculate residuals: difference between observed and predicted values
- Plot sum of squared residuals on graph
- Y-axis: sum of squared residuals
- X-axis: different slope values for fitted line
- Horizontal line slope = 0
- Test slopes: 0.2, 0.4, etc.
- Best fitting line at bottom of parabola: slope = 0.45 (lowest sum of squared residuals)
Ridge Regression Analysis (L2 Penalty)
- Ridge penalty = L2-norm (author prefers "squared penalty")
- Lambda = 0 (thick blue line):
- No penalty applied
- Original sum of squared residuals only
- Lambda = 10 (thick orange line):
- Horizontal line (slope = 0): penalty = 0
- Slope = 0.2: penalty = 0.4
- Slope = 0.4: penalty = 1.6
- Optimal slope smaller than lambda = 0 case
- Lambda = 20 (thick green line):
- Minimum value closer to zero
- Optimal slope shrinks further
- Lambda = 40 (thick purple line):
- Slope shrinks even more
- Key characteristic: As lambda increases, optimal slope approaches zero but never equals zero
- Even with lambda = 400, optimal value remains > 0
- Maintains parabola shape throughout
Lasso Regression Analysis (L1 Penalty)
- Lasso penalty = L1-norm (author prefers "absolute value penalty")
- Lambda = 0 (thick blue line):
- No penalty applied
- Original sum of squared residuals
- Lambda = 10 (thick orange line):
- Penalty activated, slope shrinking begins
- Kink appears in curve at slope = 0
- Lambda = 20 (thick green line):
- Kink at zero becomes more prominent
- Lambda = 40 (thick purple line):
- Kink at zero super obvious
- Optimal slope = 0 (lowest point in purple curve)
- Weight variable completely ignored when predicting
Key Differences Summary
- Ridge regression (L2 penalty):
- Optimal slopes shift toward zero
- Retains parabola shape
- Never reaches exactly zero (even with very high lambda)
- Lasso regression (L1 penalty):
- Optimal value shifts toward zero
- Creates kink at zero
- Can reach exactly zero optimal slope
- Results in variable selection (eliminates variables)
