AI summary by TwinMind
Power Analysis, Clearly Explained!!!
StatQuest with Josh Starmer
16m 44s
Introduction and Prerequisites
- Josh Starmer from StatQuest explaining power analysis
- Prerequisites: understanding statistical power, population vs estimated parameters, p-hacking concepts
- Focus on avoiding p-hacking through proper power analysis
Drug Comparison Example Setup
- Hypothetical virus treatment study with Drug A vs Drug B
- Initial sample: 3 people per drug group
- Drug A appeared better (faster recovery times)
- Statistical test results: p-value = 0.06
- Cannot claim Drug A better since p > 0.05 significance threshold
- Temptation to add more subjects would constitute p-hacking
Power Analysis Purpose and Definition
- Alternative to p-hacking: determine proper sample size upfront
- Power analysis determines sample size ensuring high probability of correctly rejecting null hypothesis
- Guarantees sufficient data for good decision regardless of final p-value
- Power = probability of correctly rejecting null hypothesis when alternative is true
Factors Affecting Statistical Power
- Two main factors:
- Overlap between distributions being compared
- Sample size (number of measurements per group)
- Example: For power = 0.8 (80% chance correct rejection)
- Little overlap → small sample size needed
- More overlap → larger sample size required
Sample Size Impact on Mean Estimation
- Statistical tests compare summaries (means) not individual measurements
- Single measurement estimation:
- High variation in estimated means
- 25% probability of estimates far left/right of population mean
- Low confidence in estimates
- Two measurement estimation:
- Extreme measurements have less effect
- Estimated means closer to population mean
- Higher confidence than single measurement
- Ten measurement estimation:
- Estimated means very close to population means
- Minimal overlap between group estimates
- High probability of correctly rejecting null hypothesis
Central Limit Theorem Application
- Results apply to any underlying distribution (not just normal)
- Larger sample sizes improve mean estimation accuracy regardless of data distribution
Power Analysis Implementation Steps
Step 1: Choose Desired Power Level
- Common value: 0.8 (80% probability)
- Can select any value between 0 and 1
Step 2: Set Significance Threshold (Alpha)
- Common threshold: 0.05
- Can use any value between 0 and 1
Step 3: Estimate Effect Size
- Effect size (d) = (difference in means) / (pooled standard deviation)
- Pooled standard deviation = √[(s₁² + s₂²)/2]
- Where s₁, s₂ are estimated standard deviations for each group
Example Calculation
- Original data: mean difference = 10, standard deviations = 7 and 6
- Pooled standard deviation = √[(7² + 6²)/2] = 6.5
- Effect size = 10/6.5 = 1.5
- Using online power calculator with:
- Power = 0.8
- Alpha = 0.05
- Effect size = 1.5
- Result: Sample size = 9 per group
Parameter Estimation Sources
- Prior data from similar studies
- Literature search
- Educated guesses (worst case scenario)
- Multiple methods exist for calculating effect sizes
Summary of Key Relationships
- Overlapping distributions require larger sample sizes for adequate power
- Small sample size → low confidence in estimated means → low power
- Large sample size → high confidence in estimated means → high power
- Extreme observations have less impact with larger samples
- Better mean estimates → less overlap → higher probability of correct rejection
