AI summary by TwinMind

Lecture 17: The Linear Model

MIT OpenCourseWare

1h 20m 24s

Course Context and Transition to Linear Regression

  • Review of first half semester: established probability foundation, focused on estimating parameters of univariate distributions
  • Social science requires joint distributions and conditional distributions
  • Esther's causality discussion was beginning of this - special case of joint distribution with coin flip treatment variable
  • Linear regression addresses: What if second random variable is continuous instead of binary?
  • Linear model is workhorse for estimating parameters in conditional distributions

Applications of Joint Distributions

  • Prediction: Amazon targeting ads (XKCD readers likely to buy Russian Moby Dick t-shirt)
  • Causality: Dog training example - giving treats to reduce barking (answer: no)
  • Understanding behavior: Behavioral economics study on weather influence on car purchases
    • People influenced by weather on purchase day, not just expected weather
    • Sunny day = more convertible purchases; snowy day = more SUV purchases
    • Paper: "Psychological effect of weather and car purchases" in QJE

Linear Model Definition

  • Dependent variable (Y): also called explained variable or regressand
  • Independent variable (X): also called regressor or explanatory variable
  • Error term (ε): unobserved random variable
  • Parameters: β₀ (intercept) and β₁ (slope coefficient)
  • Model allows considering mean of Y as function of X

Classical Linear Regression Model Assumptions

  1. Exogeneity: X and ε are uncorrelated
  2. Identification: Variation in X variable exists (not all X values identical)
  3. Zero mean error: E[εᵢ] = 0
  4. Homoskedasticity: E[εᵢ²] = σ² for all i (constant error variance)
  5. No serial correlation: E[εᵢεⱼ] = 0 for i ≠ j
  • Alternative stronger assumption: εᵢ ~ IID N(0, σ²)

Estimation Methods Comparison

  • Ordinary Least Squares (OLS): Minimize sum of squared residuals
  • Least Absolute Deviations: Minimize sum of absolute deviations
  • Reverse Least Squares: Different geometric approach
  • Why OLS preferred:
    • Minimum variance (most efficient) unbiased estimator
    • Maximum likelihood estimator under normality
    • Consistent and asymptotically normal
    • Has closed-form solutions (no numerical minimization needed)

OLS Estimator Properties

  • Closed-form solutions: β̂₁ = Σ(xᵢ-x̄)(yᵢ-ȳ)/Σ(xᵢ-x̄)², β̂₀ = ȳ - β̂₁x̄
  • Unbiased: E[β̂₀] = β₀, E[β̂₁] = β₁
  • Variance formulas: Functions of σ², sample size n, and variance of X
  • Comparative statics:
    • Larger σ² → larger variance of β̂
    • Larger variance of X → smaller variance of β̂
    • Larger n → smaller variance of β̂
    • Mechanical relationship between β̂₀ and β̂₁ estimates

Error Variance Estimation

  • Estimator: σ̂² = (1/(n-2))Σε̂ᵢ²
  • Uses n-2 denominator because estimating 2 parameters
  • Unbiased for σ² in linear model
  • When unknown variance replaced with estimate → t-distribution (like univariate case)

Analysis of Variance and Goodness of Fit

  • Sum of Squared Residuals (SSR): Σε̂ᵢ² (what OLS minimizes)
  • Total Sum of Squares (SST): Σ(yᵢ-ȳ)²
  • Model Sum of Squares (SSM): SST - SSR
  • R-squared: R² = 1 - SSR/SST = SSM/SST
    • Unit-free measure between 0 and 1
    • In bivariate regression, R² equals squared sample correlation coefficient
    • Generalizes to multiple regression

Hypothesis Testing

  • F-test: Tests H₀: β₁ = 0
    • Test statistic: F = (n-2)R²/(1-R²)
    • F-distributed under null with normal errors
    • Reject for large F values
  • t-tests: Individual coefficient tests (provided automatically in software)

Regression Output Interpretation

  • Stata output: Coefficient, standard error, t-statistic, p-value, F-test results
  • R output: Similar information, different layout
  • β̂₁ interpretation: Estimated effect on Y of one-unit increase in X
  • Example: MLB 2005 data - one additional win associated with 31,000 additional fans in attendance

Practical Issues

  • Dummy variables: X takes values 0 and 1
    • β̂₀ = mean of group with X=0
    • β̂₁ = difference in means between groups
    • Useful for randomized control trials, categorical characteristics
  • Nonlinearity concerns: Linear model actually very flexible
    • Can use nonlinear functions of X and Y
    • Interaction variables (products of variables)
    • Dummy variables for nonlinear shapes
  • Kernel regression alternative: More flexible but much less efficient

Introduction to Multiple Regression

  • Motivation:
    • Account for other factors affecting Y
    • Multiple potential predictors without a priori knowledge
  • Matrix notation advantages: More elegant, handles multiple variables easily
  • Basic structure:
    • Row vector Xᵢ = [1, X₁ᵢ, X₂ᵢ, ..., Xₖᵢ] for each observation
    • Column vector β = [β₀, β₁, β₂, ..., βₖ]'
    • Model: Yᵢ = Xᵢβ + εᵢ
  • Full matrix form: Y = Xβ + ε
    • Y: n×1 vector of outcomes
    • X: n×(k+1) matrix of regressors
    • β: (k+1)×1 vector of parameters
    • ε: n×1 vector of errors

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome