AI summary by TwinMind

Lecture 15: Analyzing Randomized Experiments

MIT OpenCourseWare

1h 19m 25s

Introduction to Randomized Experiment Analysis

  • Focus on completely randomized experiments: NT units to treatment, NC units to control
  • Three main analytical methods: Fisher exact test, Neyman's approach, regression methods
  • No econometric knowledge needed - basic statistics/probability sufficient
  • Sample treated as population of interest (Fisher/Neyman approach) vs. sample from larger population

Fisher's Approach: Sharp Null Hypothesis

  • Sharp null hypothesis: treatment has no effect on anybody whatsoever (not just average effect)
  • Everyone's potential outcome treated = potential outcome control
  • Solves missing data problem - under null, can recover entire counterfactual distribution
  • Can compute any statistic (max, rank, absolute difference in means)
  • Test statistic: absolute value of difference in means by treatment status

Fisher Exact Test Example: Honey for Cough Study

  • 6 kids: 3 treatment (honey), 3 control (nothing)
  • Outcome: cough severity (1-6 scale)
  • Observed data: Treatment group (3,5,0), Control group (4,0,1)
  • Difference in means = 1
  • All possible assignments: C(6,3) = 20 permutations
  • P-value = 16/20 = 0.8 (cannot reject null)
  • With larger sample, honey study showed significant effects

Fisher Test Implementation

  • For small samples: compute all possible permutations
  • For large samples: use random subset of permutations
  • Can test any sharp hypothesis (treatment effect = 0, 1, 2, etc.)
  • Can construct exact confidence intervals
  • Differs from bootstrap: keeps sample fixed, reassigns treatment

Neyman's Approach: Average Treatment Effect

  • Interested in average treatment effect: E[Yi(1)] - E[Yi(0)]
  • Allows heterogeneous treatment effects (unlike Fisher's homogeneity assumption)
  • Estimator: τ̂ = Ȳ_T - Ȳ_C (difference in sample means)

Unbiasedness Proof

  • Replace Yi_obs with WiYi(1) + (1-Wi)Yi(0)
  • E[Wi] = NT/N, E[1-Wi] = NC/N
  • E[τ̂] = average treatment effect (unbiased)

Variance Estimation

  • True variance: Var(Yi(1))/NT + Var(Yi(0))/NC - 2*Cov(Yi(1),Yi(0))/N
  • Third term (covariance) unobservable - creates estimation problem
  • Neyman's solution: ignore third term (conservative, as it's negative)

Three Justifications for Ignoring Covariance Term

  1. Conservative: overestimates variance
  2. Correct under constant treatment effects
  3. Correct for super-population inference (third term vanishes asymptotically)

Confidence Intervals and Hypothesis Testing

  • Standard confidence interval: τ̂ ± critical_value * SE(τ̂)
  • Critical values from t-distribution (small samples) or normal (large samples)
  • Neyman hypothesis test: H0: average treatment effect = 0
  • Test statistic: (Ȳ_T - Ȳ_C) / SE follows t-distribution

Empirical Example: Teacher Incentive Study in India

  • Problem: 25% teacher absenteeism rate
  • Treatment: financial incentives for attendance (camera monitoring)
  • Results:
    • Attendance: 58% control vs 80% treatment (22 percentage points)
    • 95% CI: [15, 28], t-statistic > 1.96 (significant)
    • Test scores: mixed results, some significant improvements
    • Fraction taking written test: not significantly different (good for interpretation)

Power Calculations and Sample Size Determination

  • Trade-off: larger samples = lower variance but higher cost
  • Power = 1 - β (probability of detecting true effect)
  • Type I error (α): reject true null (typically 5%)
  • Type II error (β): fail to reject false null

Required Inputs

  • α (significance level): typically 0.05
  • β (Type II error rate): typically 0.20 (80% power)
  • τ (expected treatment effect): biggest challenge
  • σ² (outcome variance): from prior data
  • γ (fraction assigned to treatment): typically 0.5

Challenges in Power Calculation

  • τ estimation problems:
    • Partners overestimate effects
    • Limited pilot data
    • No prior studies
  • Solutions:
    • Use minimum detectable effect that would prompt policy action
    • Consider treatment cost-effectiveness
    • Vary parameters to see sensitivity

Power Calculation Formula

  • Test statistic: (Ȳ_T - Ȳ_C) / SE ~ N(τ/SE, 1) under alternative
  • Required sample size: n = [Φ⁻¹(1-α/2) + Φ⁻¹(1-β)]² × σ² × [1/(γ(1-γ))] / τ²
  • Optimal allocation: γ = 0.5 when variances equal
  • Online calculators available for practical implementation

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome