AI summary by TwinMind

StatQuest: One or Two Tailed P-Values

StatQuest with Josh Starmer

7m 5s

Introduction & Setup

  • StatQuest episode from UNC Chapel Hill Genetics Department
  • Topic: One-tailed vs two-tailed statistical tests
  • Example scenario: New cancer treatment clinical trial
    • 6 patients total
    • Red dots = new treatment patients
    • Black dots = standard treatment patients
    • Values range from better to worse outcomes

Statistical Test Results Comparison

  • One-tailed test p-value: 0.03 (significant at α = 0.05)
  • Two-tailed test p-value: 0.06 (not significant at α = 0.05)
  • Data suggests new treatment performs better than standard

Test Type Definitions

  • One-tailed test: Tests hypothesis that new treatment is better than standard
  • Two-tailed test: Tests whether new treatment is better, worse, or not significantly different
  • One-tailed p-value smaller because doesn't distinguish between "worse" and "not significantly different"

P-Hacking Problem

  • Cannot decide test type after seeing data
  • Must choose statistical test and p-value threshold before experiment
  • Post-hoc test selection increases probability of bogus results

Simulation Demonstration

  • Used standard normal distribution for sampling
  • Computer picked 3 numbers from distribution center (occasional extreme values)
  • Two samples from identical distribution
  • Most two-tailed t-tests should yield p > 0.05 (samples overlap)
  • Occasional non-overlapping samples create false positives (p < 0.05)

False Positive Rate Analysis

  • 10,000 two-tailed t-tests performed
  • Expected false positives: 5% of 10,000 = 500
  • Actual results: ~500 false positives (as expected)
  • Histogram shows uniform distribution across p-value bins

P-Hacking Impact Simulation

  • Mimicked switching to one-tailed test when data looked favorable
  • Condition: Sample 1 had ≥2 values less than all Sample 2 values
  • Results: ~800 false positives
  • False positive rate increased from 5% to 8%
  • Demonstrates inflation despite using 0.05 significance threshold

Final Recommendation

  • Always want to know "both sides of the story"
  • Two-tailed tests required for complete understanding
  • Cancer treatment example: Must know if treatment improves OR worsens outcomes
  • Universal principle applies to all data with 1 vs 2-tailed options
  • Note: Not all statistical tests offer this choice

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome