AI summary by TwinMind
p-hacking: What it is and how to avoid it!
StatQuest with Josh Starmer
13m 44s
Introduction to P-Hacking
- P-hacking = misuse/abuse of analysis techniques resulting in false positives
- Assumes familiarity with p-values
- Example scenario: developing drug to reduce virus recovery time
Drug Testing Example - Initial P-Hack
- Tested multiple candidate drugs sequentially
- Drug A: no apparent improvement in recovery time (3 people per group)
- Drug B: no apparent improvement in recovery time
- Drug C: appeared effective, reduced recovery time significantly
- Calculated p-value = 0.02 (< 0.05 threshold)
- Rejected null hypothesis (no difference between groups)
- This was P-hacking - tested multiple drugs but only reported the "successful" one
Understanding the Statistical Foundation
- Recovery time distribution: normal curve fitted to large population data
- 2.5% recovered in < 5 days
- 95% recovered between 5-15 days
- 2.5% recovered in > 15 days
- Small sample sizes (3 people) create variability in group comparisons
Multiple Testing Problem Demonstration
- Repeatedly sampled 2 groups of 3 people from same distribution
- Most comparisons showed non-significant differences (p > 0.05)
- Example: p = 0.86, p = 0.63, p = 0.06
- Eventually found significant result: p = 0.02
- False positive - both samples from identical distribution
- 5% threshold means ~5% of tests on same distribution will show false significance
Multiple Testing Problem
- More tests = more false positives
- 100 tests → expect ~5 false positives (5%)
- 10,000 tests → expect ~500 false positives
- Terminology: Multiple testing problem
Solutions for Multiple Testing
- False Discovery Rate (FDR) method
- Input all p-values from all comparisons
- Outputs adjusted p-values (usually larger)
- Some false positives get adjusted p-values > 0.05
- Critical requirement: Must include ALL p-values, not cherry-picked ones
- Don't select only tests that look promising
Subtle P-Hacking: Adding Data
- Example: initial p-value = 0.06 (close to 0.05 threshold)
- Tempting to add more data when p-value nearly significant
- Added 1 measurement to each group → p-value = 0.02
- This is also P-hacking - calculated multiple p-values to make decision
- High probability of false positive when adding data to borderline results
- Theory assumes single p-value calculation per decision
Prevention: Power Analysis
- Determine proper sample size BEFORE experiment
- Power analysis performed pre-experiment
- Calculates replicates needed for high probability of correctly rejecting null hypothesis
- Prevents post-hoc data addition
Summary and Best Practices
- Multiple drug testing: Calculate p-value for each test, adjust with FDR
- Borderline results: Don't add observations to existing data
- Proper approach: Use existing data for power analysis to determine correct sample size
- Both practices help prevent false positive reporting
