AI summary by TwinMind
Lecture 15: Analyzing Randomized Experiments
MIT OpenCourseWare
1h 19m 25s
Introduction to Randomized Experiment Analysis
- Focus on completely randomized experiments: NT units to treatment, NC units to control
- Three main analytical methods: Fisher exact test, Neyman's approach, regression methods
- No econometric knowledge needed - basic statistics/probability sufficient
- Sample treated as population of interest (Fisher/Neyman approach) vs. sample from larger population
Fisher's Approach: Sharp Null Hypothesis
- Sharp null hypothesis: treatment has no effect on anybody whatsoever (not just average effect)
- Everyone's potential outcome treated = potential outcome control
- Solves missing data problem - under null, can recover entire counterfactual distribution
- Can compute any statistic (max, rank, absolute difference in means)
- Test statistic: absolute value of difference in means by treatment status
Fisher Exact Test Example: Honey for Cough Study
- 6 kids: 3 treatment (honey), 3 control (nothing)
- Outcome: cough severity (1-6 scale)
- Observed data: Treatment group (3,5,0), Control group (4,0,1)
- Difference in means = 1
- All possible assignments: C(6,3) = 20 permutations
- P-value = 16/20 = 0.8 (cannot reject null)
- With larger sample, honey study showed significant effects
Fisher Test Implementation
- For small samples: compute all possible permutations
- For large samples: use random subset of permutations
- Can test any sharp hypothesis (treatment effect = 0, 1, 2, etc.)
- Can construct exact confidence intervals
- Differs from bootstrap: keeps sample fixed, reassigns treatment
Neyman's Approach: Average Treatment Effect
- Interested in average treatment effect: E[Yi(1)] - E[Yi(0)]
- Allows heterogeneous treatment effects (unlike Fisher's homogeneity assumption)
- Estimator: τ̂ = Ȳ_T - Ȳ_C (difference in sample means)
Unbiasedness Proof
- Replace Yi_obs with WiYi(1) + (1-Wi)Yi(0)
- E[Wi] = NT/N, E[1-Wi] = NC/N
- E[τ̂] = average treatment effect (unbiased)
Variance Estimation
- True variance: Var(Yi(1))/NT + Var(Yi(0))/NC - 2*Cov(Yi(1),Yi(0))/N
- Third term (covariance) unobservable - creates estimation problem
- Neyman's solution: ignore third term (conservative, as it's negative)
Three Justifications for Ignoring Covariance Term
- Conservative: overestimates variance
- Correct under constant treatment effects
- Correct for super-population inference (third term vanishes asymptotically)
Confidence Intervals and Hypothesis Testing
- Standard confidence interval: τ̂ ± critical_value * SE(τ̂)
- Critical values from t-distribution (small samples) or normal (large samples)
- Neyman hypothesis test: H0: average treatment effect = 0
- Test statistic: (Ȳ_T - Ȳ_C) / SE follows t-distribution
Empirical Example: Teacher Incentive Study in India
- Problem: 25% teacher absenteeism rate
- Treatment: financial incentives for attendance (camera monitoring)
- Results:
- Attendance: 58% control vs 80% treatment (22 percentage points)
- 95% CI: [15, 28], t-statistic > 1.96 (significant)
- Test scores: mixed results, some significant improvements
- Fraction taking written test: not significantly different (good for interpretation)
Power Calculations and Sample Size Determination
- Trade-off: larger samples = lower variance but higher cost
- Power = 1 - β (probability of detecting true effect)
- Type I error (α): reject true null (typically 5%)
- Type II error (β): fail to reject false null
Required Inputs
- α (significance level): typically 0.05
- β (Type II error rate): typically 0.20 (80% power)
- τ (expected treatment effect): biggest challenge
- σ² (outcome variance): from prior data
- γ (fraction assigned to treatment): typically 0.5
Challenges in Power Calculation
- τ estimation problems:
- Partners overestimate effects
- Limited pilot data
- No prior studies
- Solutions:
- Use minimum detectable effect that would prompt policy action
- Consider treatment cost-effectiveness
- Vary parameters to see sensitivity
Power Calculation Formula
- Test statistic: (Ȳ_T - Ȳ_C) / SE ~ N(τ/SE, 1) under alternative
- Required sample size: n = [Φ⁻¹(1-α/2) + Φ⁻¹(1-β)]² × σ² × [1/(γ(1-γ))] / τ²
- Optimal allocation: γ = 0.5 when variances equal
- Online calculators available for practical implementation
