AI summary by TwinMind
Logistic Regression Details Pt 2: Maximum Likelihood
StatQuest with Josh Starmer
10m 23s
Introduction and Context
- Follow-up video on logistic regression series by Josh Starmer (StatQuest)
- Previous video covered logistic regression coefficients (Part 1)
- Focus: how logistic regression lines are optimized using maximum likelihood
- Comparison with linear regression fitting methods
Linear Regression Review
- Uses least squares method for line fitting
- Process: measure residuals (distances between data and line)
- Square residuals to prevent negative/positive cancellation
- Sum all squared residuals
- Rotate line iteratively to find smallest sum of squared residuals
- Line with smallest sum = best fit
Logistic Regression Setup
- Example: determining effect of weight on obesity in mice
- Data: obese mice vs non-obese mice
- Goal: draw best fitting squiggle for data
- Y-axis transformation: probability of obesity → log odds of obesity
Problem with Least Squares in Logistic Regression
- Log odds transformation pushes raw data to positive/negative infinity
- Residuals (distances from data points to line) = positive/negative infinity
- Cannot use least squares method
- Solution: maximum likelihood estimation
Maximum Likelihood Process
- Project original data points onto candidate line
- Each sample gets candidate log-odds value
- Example: log odds = 2.1, log odds = 1.4
- Transform candidate log odds to candidate probabilities
- Formula: P = e^(log odds) / (1 + e^(log odds))
Mathematical Transformation Details
- Converting probability → log odds equation to log odds → probability
- Steps:
- Exponentiate both sides
- Multiply both sides by (1 - P)
- Multiply (1 - P) and e^(log odds)
- Add P × e^(log odds) to both sides
- Factor out P on left side
- Divide both sides by (1 + e^(log odds))
Likelihood Calculations
Obese Mice Likelihoods
- Likelihood = predicted probability (y-axis value where point intersects squiggle)
- Example values: 0.49, 0.9, 0.91, 0.92
- Overall likelihood for obese mice = product of individual likelihoods
Non-obese Mice Likelihoods
- Likelihood = 1 - probability of being obese
- Examples:
- Mouse with 0.9 obesity probability → 1 - 0.9 = 0.1 likelihood not obese
- Mouse with 0.3 obesity probability → 1 - 0.3 = 0.7 likelihood not obese
- Mice with 0.01 obesity probability → 1 - 0.01 = 0.99 likelihood not obese
Log-Likelihood Calculation
- Statisticians prefer log-likelihood over likelihood
- Same squiggle maximizes both likelihood and log-likelihood
- Log-likelihood: add logs of individual likelihoods (vs multiplying likelihoods)
- Example result: log-likelihood = -3.77
Optimization Process
- Rotate line and recalculate log-likelihood
- Project data onto new line
- Transform log odds to probabilities
- Calculate new log-likelihood
- Example: second line log-likelihood = -4.15 (worse than first)
- Algorithm intelligently rotates line to increase log-likelihood
- Finds optimal fit after few rotations
- Final result: line that maximizes likelihood = best fit
Future Topics
- Need r-squared value and p-value for model evaluation
- Challenge: must calculate without usual residuals
- Next video: r-squared and p-values for logistic regression
