AI summary by TwinMind

Logistic Regression Details Pt 3: R-squared and p-value

StatQuest with Josh Starmer

15m 25s

Introduction to R-squared Problem in Logistic Regression

  • No consensus on calculating R-squared for logistic regression (10+ different methods)
  • Linear models have agreed-upon R-squared calculation, logistic regression does not
  • Focus on McFadden's pseudo R-squared - commonly used, easily calculated from R output
  • Similar methodology to linear model R-squared calculation

Linear Regression R-squared Review

  • R-squared calculated using residuals in linear models
  • SS fit = sum of squares of residuals around best fitting line
  • SS mean = sum of squared residuals around worst fitting line (mean of y-axis values)
  • R-squared formula: (SS mean - SS fit) / SS mean
  • R-squared = percentage of variation around mean eliminated by fitting line
  • Range: 0 to 1
    • R-squared = 0 when SS fit = SS mean (no relationship)
    • R-squared = 1 when SS fit = 0 (perfect fit)

Logistic Regression R-squared Challenges

  • Residuals for logistic regression are infinite - cannot use them
  • Need alternative approach using log likelihood instead of residuals
  • Project data onto best fitting line, translate log odds back to probabilities
  • Calculate log likelihood of data given best fitting curve

McFadden's Pseudo R-squared Calculation Method

LL fit (substitute for SS fit)

  • Log likelihood of fitted line = -3.77 (example value)
  • Measure of good fit using best fitting logistic curve

LL overall probability (substitute for SS mean)

  • Calculate overall log odds of obesity without considering weight
  • Total obese mice / total non-obese mice, then take log
  • Results in horizontal line at 0.22 log odds
  • Translates to probability = 0.56
  • Alternative: calculate overall probability of obesity directly
  • LL overall probability = -6.18 (example value)

R-squared Formula

  • McFadden's pseudo R-squared = (LL overall - LL fit) / LL overall
  • Example calculation: (-6.18 - (-3.77)) / (-6.18) = 0.39

Validation of Log Likelihood R-squared Range

Poor Fit Scenario (R-squared = 0)

  • Light mice: both obese and non-obese
  • Heavy mice: both obese and non-obese
  • Best fitting line: intercept = -0.22, slope ≈ 0 (5×10^-17)
  • LL fit = LL overall probability = -6.18
  • R-squared = 0

Perfect Fit Scenario (R-squared = 1)

  • Maximum likelihood line: intercept = -63.72, slope = 22.42
  • LL fit = 0 (log of 1 = 0, sum of zeros)
  • LL overall probability = -6.18
  • R-squared = 1

Log Likelihood Properties

  • Always between 0 and negative infinity
  • Good fits: log likelihood close to 0
  • Bad fits: larger negative log likelihoods
  • Taking logs of values between 0 and 1

P-value Calculation

  • Formula: 2 × (LL fit - LL overall probability) = chi-squared value
  • Degrees of freedom = difference in number of parameters between models
    • LL fit: 2 parameters (intercept + slope)
    • LL overall probability: 1 parameter (intercept only)
    • df = 1
  • Worst case: LL fit = LL overall probability → chi-squared = 0, p-value = 1
  • Typical case: LL fit closer to zero → positive chi-squared value, smaller p-value
  • Example: chi-squared = 4.82, p-value = 0.03
  • Conclusion: relationship not due to chance, R-squared = 0.39 shows effect size

Technical Notes on Formula Presentation

  • Wild formulas include saturated model terms
  • Saturated model log likelihood = 0 in logistic regression (can omit)
  • Saturated model included for compatibility with other generalized linear models
  • Simple equations presented in this explanation work for logistic regression specifically

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome