AI summary by TwinMind
Logistic Regression Details Pt 3: R-squared and p-value
StatQuest with Josh Starmer
15m 25s
Introduction to R-squared Problem in Logistic Regression
- No consensus on calculating R-squared for logistic regression (10+ different methods)
- Linear models have agreed-upon R-squared calculation, logistic regression does not
- Focus on McFadden's pseudo R-squared - commonly used, easily calculated from R output
- Similar methodology to linear model R-squared calculation
Linear Regression R-squared Review
- R-squared calculated using residuals in linear models
- SS fit = sum of squares of residuals around best fitting line
- SS mean = sum of squared residuals around worst fitting line (mean of y-axis values)
- R-squared formula: (SS mean - SS fit) / SS mean
- R-squared = percentage of variation around mean eliminated by fitting line
- Range: 0 to 1
- R-squared = 0 when SS fit = SS mean (no relationship)
- R-squared = 1 when SS fit = 0 (perfect fit)
Logistic Regression R-squared Challenges
- Residuals for logistic regression are infinite - cannot use them
- Need alternative approach using log likelihood instead of residuals
- Project data onto best fitting line, translate log odds back to probabilities
- Calculate log likelihood of data given best fitting curve
McFadden's Pseudo R-squared Calculation Method
LL fit (substitute for SS fit)
- Log likelihood of fitted line = -3.77 (example value)
- Measure of good fit using best fitting logistic curve
LL overall probability (substitute for SS mean)
- Calculate overall log odds of obesity without considering weight
- Total obese mice / total non-obese mice, then take log
- Results in horizontal line at 0.22 log odds
- Translates to probability = 0.56
- Alternative: calculate overall probability of obesity directly
- LL overall probability = -6.18 (example value)
R-squared Formula
- McFadden's pseudo R-squared = (LL overall - LL fit) / LL overall
- Example calculation: (-6.18 - (-3.77)) / (-6.18) = 0.39
Validation of Log Likelihood R-squared Range
Poor Fit Scenario (R-squared = 0)
- Light mice: both obese and non-obese
- Heavy mice: both obese and non-obese
- Best fitting line: intercept = -0.22, slope ≈ 0 (5×10^-17)
- LL fit = LL overall probability = -6.18
- R-squared = 0
Perfect Fit Scenario (R-squared = 1)
- Maximum likelihood line: intercept = -63.72, slope = 22.42
- LL fit = 0 (log of 1 = 0, sum of zeros)
- LL overall probability = -6.18
- R-squared = 1
Log Likelihood Properties
- Always between 0 and negative infinity
- Good fits: log likelihood close to 0
- Bad fits: larger negative log likelihoods
- Taking logs of values between 0 and 1
P-value Calculation
- Formula: 2 × (LL fit - LL overall probability) = chi-squared value
- Degrees of freedom = difference in number of parameters between models
- LL fit: 2 parameters (intercept + slope)
- LL overall probability: 1 parameter (intercept only)
- df = 1
- Worst case: LL fit = LL overall probability → chi-squared = 0, p-value = 1
- Typical case: LL fit closer to zero → positive chi-squared value, smaller p-value
- Example: chi-squared = 4.82, p-value = 0.03
- Conclusion: relationship not due to chance, R-squared = 0.39 shows effect size
Technical Notes on Formula Presentation
- Wild formulas include saturated model terms
- Saturated model log likelihood = 0 in logistic regression (can omit)
- Saturated model included for compatibility with other generalized linear models
- Simple equations presented in this explanation work for logistic regression specifically
