AI summary by TwinMind
Lecture 17: The Linear Model
MIT OpenCourseWare
1h 20m 24s
Course Context and Transition to Linear Regression
- Review of first half semester: established probability foundation, focused on estimating parameters of univariate distributions
- Social science requires joint distributions and conditional distributions
- Esther's causality discussion was beginning of this - special case of joint distribution with coin flip treatment variable
- Linear regression addresses: What if second random variable is continuous instead of binary?
- Linear model is workhorse for estimating parameters in conditional distributions
Applications of Joint Distributions
- Prediction: Amazon targeting ads (XKCD readers likely to buy Russian Moby Dick t-shirt)
- Causality: Dog training example - giving treats to reduce barking (answer: no)
- Understanding behavior: Behavioral economics study on weather influence on car purchases
- People influenced by weather on purchase day, not just expected weather
- Sunny day = more convertible purchases; snowy day = more SUV purchases
- Paper: "Psychological effect of weather and car purchases" in QJE
Linear Model Definition
- Dependent variable (Y): also called explained variable or regressand
- Independent variable (X): also called regressor or explanatory variable
- Error term (ε): unobserved random variable
- Parameters: β₀ (intercept) and β₁ (slope coefficient)
- Model allows considering mean of Y as function of X
Classical Linear Regression Model Assumptions
- Exogeneity: X and ε are uncorrelated
- Identification: Variation in X variable exists (not all X values identical)
- Zero mean error: E[εᵢ] = 0
- Homoskedasticity: E[εᵢ²] = σ² for all i (constant error variance)
- No serial correlation: E[εᵢεⱼ] = 0 for i ≠ j
- Alternative stronger assumption: εᵢ ~ IID N(0, σ²)
Estimation Methods Comparison
- Ordinary Least Squares (OLS): Minimize sum of squared residuals
- Least Absolute Deviations: Minimize sum of absolute deviations
- Reverse Least Squares: Different geometric approach
- Why OLS preferred:
- Minimum variance (most efficient) unbiased estimator
- Maximum likelihood estimator under normality
- Consistent and asymptotically normal
- Has closed-form solutions (no numerical minimization needed)
OLS Estimator Properties
- Closed-form solutions: β̂₁ = Σ(xᵢ-x̄)(yᵢ-ȳ)/Σ(xᵢ-x̄)², β̂₀ = ȳ - β̂₁x̄
- Unbiased: E[β̂₀] = β₀, E[β̂₁] = β₁
- Variance formulas: Functions of σ², sample size n, and variance of X
- Comparative statics:
- Larger σ² → larger variance of β̂
- Larger variance of X → smaller variance of β̂
- Larger n → smaller variance of β̂
- Mechanical relationship between β̂₀ and β̂₁ estimates
Error Variance Estimation
- Estimator: σ̂² = (1/(n-2))Σε̂ᵢ²
- Uses n-2 denominator because estimating 2 parameters
- Unbiased for σ² in linear model
- When unknown variance replaced with estimate → t-distribution (like univariate case)
Analysis of Variance and Goodness of Fit
- Sum of Squared Residuals (SSR): Σε̂ᵢ² (what OLS minimizes)
- Total Sum of Squares (SST): Σ(yᵢ-ȳ)²
- Model Sum of Squares (SSM): SST - SSR
- R-squared: R² = 1 - SSR/SST = SSM/SST
- Unit-free measure between 0 and 1
- In bivariate regression, R² equals squared sample correlation coefficient
- Generalizes to multiple regression
Hypothesis Testing
- F-test: Tests H₀: β₁ = 0
- Test statistic: F = (n-2)R²/(1-R²)
- F-distributed under null with normal errors
- Reject for large F values
- t-tests: Individual coefficient tests (provided automatically in software)
Regression Output Interpretation
- Stata output: Coefficient, standard error, t-statistic, p-value, F-test results
- R output: Similar information, different layout
- β̂₁ interpretation: Estimated effect on Y of one-unit increase in X
- Example: MLB 2005 data - one additional win associated with 31,000 additional fans in attendance
Practical Issues
- Dummy variables: X takes values 0 and 1
- β̂₀ = mean of group with X=0
- β̂₁ = difference in means between groups
- Useful for randomized control trials, categorical characteristics
- Nonlinearity concerns: Linear model actually very flexible
- Can use nonlinear functions of X and Y
- Interaction variables (products of variables)
- Dummy variables for nonlinear shapes
- Kernel regression alternative: More flexible but much less efficient
Introduction to Multiple Regression
- Motivation:
- Account for other factors affecting Y
- Multiple potential predictors without a priori knowledge
- Matrix notation advantages: More elegant, handles multiple variables easily
- Basic structure:
- Row vector Xᵢ = [1, X₁ᵢ, X₂ᵢ, ..., Xₖᵢ] for each observation
- Column vector β = [β₀, β₁, β₂, ..., βₖ]'
- Model: Yᵢ = Xᵢβ + εᵢ
- Full matrix form: Y = Xβ + ε
- Y: n×1 vector of outcomes
- X: n×(k+1) matrix of regressors
- β: (k+1)×1 vector of parameters
- ε: n×1 vector of errors
