AI summary by TwinMind

p-hacking: What it is and how to avoid it!

StatQuest with Josh Starmer

13m 44s

Introduction to P-Hacking

  • P-hacking = misuse/abuse of analysis techniques resulting in false positives
  • Assumes familiarity with p-values
  • Example scenario: developing drug to reduce virus recovery time

Drug Testing Example - Initial P-Hack

  • Tested multiple candidate drugs sequentially
  • Drug A: no apparent improvement in recovery time (3 people per group)
  • Drug B: no apparent improvement in recovery time
  • Drug C: appeared effective, reduced recovery time significantly
  • Calculated p-value = 0.02 (< 0.05 threshold)
  • Rejected null hypothesis (no difference between groups)
  • This was P-hacking - tested multiple drugs but only reported the "successful" one

Understanding the Statistical Foundation

  • Recovery time distribution: normal curve fitted to large population data
    • 2.5% recovered in < 5 days
    • 95% recovered between 5-15 days
    • 2.5% recovered in > 15 days
  • Small sample sizes (3 people) create variability in group comparisons

Multiple Testing Problem Demonstration

  • Repeatedly sampled 2 groups of 3 people from same distribution
  • Most comparisons showed non-significant differences (p > 0.05)
    • Example: p = 0.86, p = 0.63, p = 0.06
  • Eventually found significant result: p = 0.02
  • False positive - both samples from identical distribution
  • 5% threshold means ~5% of tests on same distribution will show false significance

Multiple Testing Problem

  • More tests = more false positives
  • 100 tests → expect ~5 false positives (5%)
  • 10,000 tests → expect ~500 false positives
  • Terminology: Multiple testing problem

Solutions for Multiple Testing

  • False Discovery Rate (FDR) method
    • Input all p-values from all comparisons
    • Outputs adjusted p-values (usually larger)
    • Some false positives get adjusted p-values > 0.05
  • Critical requirement: Must include ALL p-values, not cherry-picked ones
  • Don't select only tests that look promising

Subtle P-Hacking: Adding Data

  • Example: initial p-value = 0.06 (close to 0.05 threshold)
  • Tempting to add more data when p-value nearly significant
  • Added 1 measurement to each group → p-value = 0.02
  • This is also P-hacking - calculated multiple p-values to make decision
  • High probability of false positive when adding data to borderline results
  • Theory assumes single p-value calculation per decision

Prevention: Power Analysis

  • Determine proper sample size BEFORE experiment
  • Power analysis performed pre-experiment
  • Calculates replicates needed for high probability of correctly rejecting null hypothesis
  • Prevents post-hoc data addition

Summary and Best Practices

  • Multiple drug testing: Calculate p-value for each test, adjust with FDR
  • Borderline results: Don't add observations to existing data
  • Proper approach: Use existing data for power analysis to determine correct sample size
  • Both practices help prevent false positive reporting

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome