AI summary by TwinMind
p-hacking and power calculations
StatQuest with Josh Starmer
19m 11s
Introduction and Scenario Setup
- StatQuest educational video from UNC Chapel Hill Genetics Department
- Topic: P-hacking pitfalls and power analysis prevention
- Common scenario: Late night grant deadline, experiments complete, P-value = 0.051
- Question: Should you run one more replicate to get P < 0.05?
P-Value Fundamentals and False Positives
- Normal distribution example using inbred laboratory mouse strain weights
- T-test theory: 95% of time P-value should be > 0.05 when comparing samples from same population
- False positives: P-values < 0.05 when no real difference exists
- Generated 1,000 datasets from normal distribution
- Results: 53 false positives out of 1,000 tests = 5.3% (expected rate)
- Focus on borderline P-values (0.051) that cause late-night anxiety
P-Hacking Demonstration
- Experiment: Added one random value to datasets with P-values between 0.05-0.1
- Critical finding: 30% of new T-tests resulted in P < 0.05
- This represents huge false positive rate when adding samples to achieve significance
- Moral: Don't add samples until you get good P-value
Power Analysis Introduction
- Definition: Power = probability a test will correctly give small P-value
- Alternative to p-hacking: Do power calculation before experiment
- Power calculation determines required sample size in advance
- Can also use existing data for retrospective power analysis
Four Factors Affecting Statistical Power
1. Effect Size
- Large effect size: Distributions clearly separated, high power with small samples
- Small effect size: Overlapping distributions, requires larger samples
- Examples shown with mouse weight distributions (diet vs no diet)
2. Variation in Data
- Low variation: High power even with small effect sizes
- High variation: Low power, requires compensation with larger samples
- Variation directly impacts ability to detect true differences
3. Sample Size
- Large sample size compensates for small effect size and high variation
- T-tests compare means estimated from samples
- Sample size affects confidence in mean estimates
- Demonstrations with n=1, n=2, n=5, n=8, n=101
- Larger samples create more reliable mean estimates
- Standard error = standard deviation / √(sample size)
4. Statistical Test Choice
- Different tests have different power levels
- T-test is surprisingly powerful and commonly used
- Good news: T-test performs well for most applications
Power Calculation Methodology
- Requires preliminary data or educated guess
- Sources: old data, publicly available similar data, previous experiments
- Use preliminary data to estimate means and standard deviations
- Calculate effect size and variation parameters
Power Calculation Process
- Need 2 of 3 variables: effect size, variation, sample size
- With preliminary data: Estimate effect size and variation → calculate n
- Without preliminary data:
- Start with 3 replicates assumption
- Assume 2-fold difference target
- Determine required variation for good power
- Adjust sample size or effect size if variation unrealistic
RNA-seq Specific Guidance
- Biological variation already established for human data and inbred species
- Can plug in known variation values
- Either: set effect size → determine sample size, or set sample size → determine detectable effect size
Software and Tools
- Never calculate power by hand
- Websites and programs available for calculations
- Tools calculate n, effect size, or variation automatically
Resolution to Opening Scenario
- P-value = 0.051 situation resolution
- Correct approach: Use results as preliminary data for proper power calculation
- Report P-value as "suggestive" rather than trying to achieve significance
- Calculate proper sample size for next experiment
- Avoid p-hacking temptation
