Introduction to Linear Regression

• StatQuest educational video from UNC Chapel Hill genetics department
• Covers three main concepts of linear regression:
  • Use least squares to fit line to data
  • Calculate R-squared
  • Calculate p-value for R-squared

Least Squares Method Review

• Example dataset: mouse weight vs mouse size
• Goal: use mouse weight to predict mouse size
• Process:
  • Draw line through data
  • Measure distance from line to data points (called residuals)
  • Square each distance and sum them up
  • Rotate line incrementally
  • Calculate sum of squared residuals for each rotation
  • Find rotation with least sum of squares
• Result: least squares fitted line with equation parameters
  • Y-intercept parameter
  • Slope parameter
• Non-zero slope indicates weight helps predict size

R-Squared Calculation and Interpretation

• Measures how good the prediction will be
• Process:
  • Calculate average mouse size
  • Calculate sum of squares around mean (SS_mean)
  • Calculate sum of squares around fitted line (SS_fit)
• Terminology:
  • Variation around mean = (data - mean)² / n
  • Variation around fit = (data - line)² / n
  • Variance = sum of squares / sample size
• R-squared formula: (Variation around mean - Variation around fit) / Variation around mean
• Example calculation:
  • Variation around mean = 11.1
  • Variation around fit = 4.4
  • R-squared = 0.6 = 60%
• Alternative using sum of squares:
  • SS_mean = 100, SS_fit = 40
  • Same result: R-squared = 0.6

R-Squared Examples and Interpretation

• Perfect prediction example: R-squared = 1.0 (100%)
  • Variation around fit = 0 (no residuals)
• No relationship example: R-squared = 0 (0%)
  • Variation around fit = variation around mean
• R-squared applies to any equation complexity
• General process: measure distances to mean, measure distances to equation, calculate ratio

Multiple Parameter Example

• 3D example: mouse weight + tail length predicting body length
• Least squares fits plane instead of line
• Equation: Y = y-intercept + 0.7×weight + 0.5×tail_length
• Three parameters estimated by least squares
• Useless parameters automatically set to zero by least squares
• More parameters never make sum of squares worse
• Random chance can create spurious improvements
• Solution: adjusted R-squared scales by number of parameters

Statistical Significance Problem

• Two data points always give R-squared = 100%
• Any two random points can be connected by straight line
• Need p-value to determine if R-squared is statistically significant

F-Statistic and P-Value Calculation

• F-statistic formula: (Variation explained by weight) / (Variation not explained by weight)
• Numerator same as R-squared numerator
• Denominator: residual variation after fitting
• Mathematical formulation includes degrees of freedom:
  • P_fit = number of parameters in fitted line
  • P_mean = number of parameters in mean line (always 1)
  • Simple line: P_fit = 2 (y-intercept + slope)
  • P_fit - P_mean = extra parameters in fit
• Degrees of freedom adjust for parameter complexity
• More parameters require more data for reliable estimates

P-Value Generation Process

• Conceptual approach:
  • Generate random datasets
  • Calculate mean and SS_mean for each
  • Calculate fit and SS_fit for each
  • Compute F-statistic for each random dataset
  • Create histogram of F-values from random data
  • Compare actual F-value to distribution
• P-value = proportion of random F-values ≥ observed F-value
• In practice: use standard F-distribution curves
• Degrees of freedom determine distribution shape
• Smaller sample sizes relative to parameters = faster tapering distribution = smaller p-values

Summary and Requirements

• Linear regression quantifies relationships in data
• Need large R-squared (strong relationship)
• Need small p-value (reliable relationship)
• Both required for interesting/significant results