AI summary by TwinMind

Ridge vs Lasso Regression, Visualized!!!

StatQuest with Josh Starmer

9m 5s

Introduction & Setup

  • StatQuest tutorial by Josh Starmer comparing Ridge vs Lasso regression
  • Assumes prior familiarity with both regression types
  • Uses simple dataset: weight and height measurements
  • Demonstrates differences through visualization

Baseline Model & Residuals Calculation

  • Start with horizontal line (terrible fit) to data
  • Calculate residuals: difference between observed and predicted values
  • Plot sum of squared residuals on graph
    • Y-axis: sum of squared residuals
    • X-axis: different slope values for fitted line
  • Horizontal line slope = 0
  • Test slopes: 0.2, 0.4, etc.
  • Best fitting line at bottom of parabola: slope = 0.45 (lowest sum of squared residuals)

Ridge Regression Analysis (L2 Penalty)

  • Ridge penalty = L2-norm (author prefers "squared penalty")
  • Lambda = 0 (thick blue line):
    • No penalty applied
    • Original sum of squared residuals only
  • Lambda = 10 (thick orange line):
    • Horizontal line (slope = 0): penalty = 0
    • Slope = 0.2: penalty = 0.4
    • Slope = 0.4: penalty = 1.6
    • Optimal slope smaller than lambda = 0 case
  • Lambda = 20 (thick green line):
    • Minimum value closer to zero
    • Optimal slope shrinks further
  • Lambda = 40 (thick purple line):
    • Slope shrinks even more
  • Key characteristic: As lambda increases, optimal slope approaches zero but never equals zero
  • Even with lambda = 400, optimal value remains > 0
  • Maintains parabola shape throughout

Lasso Regression Analysis (L1 Penalty)

  • Lasso penalty = L1-norm (author prefers "absolute value penalty")
  • Lambda = 0 (thick blue line):
    • No penalty applied
    • Original sum of squared residuals
  • Lambda = 10 (thick orange line):
    • Penalty activated, slope shrinking begins
    • Kink appears in curve at slope = 0
  • Lambda = 20 (thick green line):
    • Kink at zero becomes more prominent
  • Lambda = 40 (thick purple line):
    • Kink at zero super obvious
    • Optimal slope = 0 (lowest point in purple curve)
    • Weight variable completely ignored when predicting

Key Differences Summary

  • Ridge regression (L2 penalty):
    • Optimal slopes shift toward zero
    • Retains parabola shape
    • Never reaches exactly zero (even with very high lambda)
  • Lasso regression (L1 penalty):
    • Optimal value shifts toward zero
    • Creates kink at zero
    • Can reach exactly zero optimal slope
    • Results in variable selection (eliminates variables)

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome