AI summary by TwinMind
StatQuest: One or Two Tailed P-Values
StatQuest with Josh Starmer
7m 5s
Introduction & Setup
- StatQuest episode from UNC Chapel Hill Genetics Department
- Topic: One-tailed vs two-tailed statistical tests
- Example scenario: New cancer treatment clinical trial
- 6 patients total
- Red dots = new treatment patients
- Black dots = standard treatment patients
- Values range from better to worse outcomes
Statistical Test Results Comparison
- One-tailed test p-value: 0.03 (significant at α = 0.05)
- Two-tailed test p-value: 0.06 (not significant at α = 0.05)
- Data suggests new treatment performs better than standard
Test Type Definitions
- One-tailed test: Tests hypothesis that new treatment is better than standard
- Two-tailed test: Tests whether new treatment is better, worse, or not significantly different
- One-tailed p-value smaller because doesn't distinguish between "worse" and "not significantly different"
P-Hacking Problem
- Cannot decide test type after seeing data
- Must choose statistical test and p-value threshold before experiment
- Post-hoc test selection increases probability of bogus results
Simulation Demonstration
- Used standard normal distribution for sampling
- Computer picked 3 numbers from distribution center (occasional extreme values)
- Two samples from identical distribution
- Most two-tailed t-tests should yield p > 0.05 (samples overlap)
- Occasional non-overlapping samples create false positives (p < 0.05)
False Positive Rate Analysis
- 10,000 two-tailed t-tests performed
- Expected false positives: 5% of 10,000 = 500
- Actual results: ~500 false positives (as expected)
- Histogram shows uniform distribution across p-value bins
P-Hacking Impact Simulation
- Mimicked switching to one-tailed test when data looked favorable
- Condition: Sample 1 had ≥2 values less than all Sample 2 values
- Results: ~800 false positives
- False positive rate increased from 5% to 8%
- Demonstrates inflation despite using 0.05 significance threshold
Final Recommendation
- Always want to know "both sides of the story"
- Two-tailed tests required for complete understanding
- Cancer treatment example: Must know if treatment improves OR worsens outcomes
- Universal principle applies to all data with 1 vs 2-tailed options
- Note: Not all statistical tests offer this choice
