AI summary by TwinMind
Ridge, Lasso and Elastic-Net Regression in R
StatQuest with Josh Starmer
17m 51s
Introduction & GLMnet Overview
- Elastic regression tutorial using GLMnet library in R
- GLMnet = Generalized Linear Models (GLM) + elastic net
- Supports linear regression, logistic regression, other models
- GLMnet parameterization differs from standard elastic net formulation
GLMnet Parameter System
- Uses single lambda + alpha instead of two separate lambdas
- Alpha controls penalty mixture:
- Alpha = 0: pure Ridge regression (lasso penalty = 0)
- Alpha = 1: pure Lasso regression (ridge penalty = 0)
- Alpha between 0-1: mixture of both penalties
- Lambda controls overall penalty strength:
- Lambda = 0: standard least squares/maximum likelihood
- Lambda > 0: elastic net penalty applied, shrinks parameter estimates
Synthetic Dataset Creation
- N = 1,000 samples, P = 5,000 parameters
- Only 15 parameters truly predictive, remaining 4,985 = random noise
- Matrix X: 1,000 × 5,000 from standard normal distribution (mean=0, sd=1)
- Vector Y: sum of first 15 columns + noise from standard normal
- Data split: 66% training (0.66 × N), 34% testing
Ridge Regression Implementation (Alpha = 0)
- cv.glmnet() function with 10-fold cross-validation
- Parameters:
- type.measure = "MSE" (mean squared error)
- alpha = 0 (Ridge regression)
- family = "gaussian" (linear regression)
- Lambda selection: lambda.1se (simplest model within 1 SE of minimum)
- Results: MSE = 14.47 on test set
Lasso Regression Implementation (Alpha = 1)
- Same cv.glmnet() setup, alpha = 1
- Results: MSE = 1.19 (much better than Ridge)
- Demonstrates Lasso's superiority for sparse data
Elastic Net Implementation (Alpha = 0.5)
- Alpha = 0.5 (50/50 Ridge/Lasso mixture)
- Results: MSE = 1.24 (slightly worse than pure Lasso)
Comprehensive Alpha Testing
- Loop through alpha values: 0, 0.1, 0.2, ..., 1.0
- Store all fits in list_of_fits
- Calculate MSE for each alpha value on test set
- Results show alpha = 1 (Lasso) performs best for this dataset
- Slight variations due to random initialization of parameters
Key Technical Notes
- cv.glmnet() requires separate X and Y inputs (no formula notation)
- For logistic regression: family = "binomial", type.measure = "deviance"
- set.seed() recommended for reproducible results
- lambda.1se preferred over lambda.min for simpler models
