AI summary by TwinMind

p-hacking and power calculations

StatQuest with Josh Starmer

19m 11s

Introduction and Scenario Setup

  • StatQuest educational video from UNC Chapel Hill Genetics Department
  • Topic: P-hacking pitfalls and power analysis prevention
  • Common scenario: Late night grant deadline, experiments complete, P-value = 0.051
  • Question: Should you run one more replicate to get P < 0.05?

P-Value Fundamentals and False Positives

  • Normal distribution example using inbred laboratory mouse strain weights
  • T-test theory: 95% of time P-value should be > 0.05 when comparing samples from same population
  • False positives: P-values < 0.05 when no real difference exists
  • Generated 1,000 datasets from normal distribution
  • Results: 53 false positives out of 1,000 tests = 5.3% (expected rate)
  • Focus on borderline P-values (0.051) that cause late-night anxiety

P-Hacking Demonstration

  • Experiment: Added one random value to datasets with P-values between 0.05-0.1
  • Critical finding: 30% of new T-tests resulted in P < 0.05
  • This represents huge false positive rate when adding samples to achieve significance
  • Moral: Don't add samples until you get good P-value

Power Analysis Introduction

  • Definition: Power = probability a test will correctly give small P-value
  • Alternative to p-hacking: Do power calculation before experiment
  • Power calculation determines required sample size in advance
  • Can also use existing data for retrospective power analysis

Four Factors Affecting Statistical Power

1. Effect Size

  • Large effect size: Distributions clearly separated, high power with small samples
  • Small effect size: Overlapping distributions, requires larger samples
  • Examples shown with mouse weight distributions (diet vs no diet)

2. Variation in Data

  • Low variation: High power even with small effect sizes
  • High variation: Low power, requires compensation with larger samples
  • Variation directly impacts ability to detect true differences

3. Sample Size

  • Large sample size compensates for small effect size and high variation
  • T-tests compare means estimated from samples
  • Sample size affects confidence in mean estimates
  • Demonstrations with n=1, n=2, n=5, n=8, n=101
  • Larger samples create more reliable mean estimates
  • Standard error = standard deviation / √(sample size)

4. Statistical Test Choice

  • Different tests have different power levels
  • T-test is surprisingly powerful and commonly used
  • Good news: T-test performs well for most applications

Power Calculation Methodology

  • Requires preliminary data or educated guess
  • Sources: old data, publicly available similar data, previous experiments
  • Use preliminary data to estimate means and standard deviations
  • Calculate effect size and variation parameters

Power Calculation Process

  • Need 2 of 3 variables: effect size, variation, sample size
  • With preliminary data: Estimate effect size and variation → calculate n
  • Without preliminary data:
    • Start with 3 replicates assumption
    • Assume 2-fold difference target
    • Determine required variation for good power
    • Adjust sample size or effect size if variation unrealistic

RNA-seq Specific Guidance

  • Biological variation already established for human data and inbred species
  • Can plug in known variation values
  • Either: set effect size → determine sample size, or set sample size → determine detectable effect size

Software and Tools

  • Never calculate power by hand
  • Websites and programs available for calculations
  • Tools calculate n, effect size, or variation automatically

Resolution to Opening Scenario

  • P-value = 0.051 situation resolution
  • Correct approach: Use results as preliminary data for proper power calculation
  • Report P-value as "suggestive" rather than trying to achieve significance
  • Calculate proper sample size for next experiment
  • Avoid p-hacking temptation

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome