AI summary by TwinMind

Power Analysis, Clearly Explained!!!

StatQuest with Josh Starmer

16m 44s

Introduction and Prerequisites

  • Josh Starmer from StatQuest explaining power analysis
  • Prerequisites: understanding statistical power, population vs estimated parameters, p-hacking concepts
  • Focus on avoiding p-hacking through proper power analysis

Drug Comparison Example Setup

  • Hypothetical virus treatment study with Drug A vs Drug B
  • Initial sample: 3 people per drug group
  • Drug A appeared better (faster recovery times)
  • Statistical test results: p-value = 0.06
  • Cannot claim Drug A better since p > 0.05 significance threshold
  • Temptation to add more subjects would constitute p-hacking

Power Analysis Purpose and Definition

  • Alternative to p-hacking: determine proper sample size upfront
  • Power analysis determines sample size ensuring high probability of correctly rejecting null hypothesis
  • Guarantees sufficient data for good decision regardless of final p-value
  • Power = probability of correctly rejecting null hypothesis when alternative is true

Factors Affecting Statistical Power

  • Two main factors:
    • Overlap between distributions being compared
    • Sample size (number of measurements per group)
  • Example: For power = 0.8 (80% chance correct rejection)
    • Little overlap → small sample size needed
    • More overlap → larger sample size required

Sample Size Impact on Mean Estimation

  • Statistical tests compare summaries (means) not individual measurements
  • Single measurement estimation:
    • High variation in estimated means
    • 25% probability of estimates far left/right of population mean
    • Low confidence in estimates
  • Two measurement estimation:
    • Extreme measurements have less effect
    • Estimated means closer to population mean
    • Higher confidence than single measurement
  • Ten measurement estimation:
    • Estimated means very close to population means
    • Minimal overlap between group estimates
    • High probability of correctly rejecting null hypothesis

Central Limit Theorem Application

  • Results apply to any underlying distribution (not just normal)
  • Larger sample sizes improve mean estimation accuracy regardless of data distribution

Power Analysis Implementation Steps

Step 1: Choose Desired Power Level

  • Common value: 0.8 (80% probability)
  • Can select any value between 0 and 1

Step 2: Set Significance Threshold (Alpha)

  • Common threshold: 0.05
  • Can use any value between 0 and 1

Step 3: Estimate Effect Size

  • Effect size (d) = (difference in means) / (pooled standard deviation)
  • Pooled standard deviation = √[(s₁² + s₂²)/2]
  • Where s₁, s₂ are estimated standard deviations for each group

Example Calculation

  • Original data: mean difference = 10, standard deviations = 7 and 6
  • Pooled standard deviation = √[(7² + 6²)/2] = 6.5
  • Effect size = 10/6.5 = 1.5
  • Using online power calculator with:
    • Power = 0.8
    • Alpha = 0.05
    • Effect size = 1.5
  • Result: Sample size = 9 per group

Parameter Estimation Sources

  • Prior data from similar studies
  • Literature search
  • Educated guesses (worst case scenario)
  • Multiple methods exist for calculating effect sizes

Summary of Key Relationships

  • Overlapping distributions require larger sample sizes for adequate power
  • Small sample size → low confidence in estimated means → low power
  • Large sample size → high confidence in estimated means → high power
  • Extreme observations have less impact with larger samples
  • Better mean estimates → less overlap → higher probability of correct rejection

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome