AI summary by TwinMind

p-values: What they are and how to interpret them

StatQuest with Josh Starmer

11m 21s

Introduction to P-Values

  • Josh Starmer introduces p-values concept using drug comparison example
  • Focus on what p-values are and how to interpret them

Drug Testing Scenario Setup

  • Comparing Drug A vs Drug B effectiveness
  • Initial test: 1 person each drug
    • Drug A: 1 person cured
    • Drug B: 1 person not cured
  • Cannot conclude Drug A better - too many confounding factors:
    • Bad medication interactions
    • Rare allergies
    • Missed doses
    • Placebo effects

Expanding Sample Size

  • Second test: 2 people per drug
    • Drug A: both people cured
    • Drug B: 1 cured, 1 not cured
  • Still inconclusive due to potential random factors
    • Mislabeled drugs
    • Individual variations

Large Scale Results - Clear Difference

  • Drug A results: 1043 cured, 3 not cured (99.7% cure rate, 1046 total)
  • Drug B results: 2 cured, 1432 not cured (0.1% cure rate, 1434 total)
  • Clear superiority of Drug A
  • Too many people cured by Drug A, too few by Drug B for random chance explanation

Ambiguous Results Scenario

  • Drug A: 37% cure rate
  • Drug B: 31% cure rate
  • Smaller difference raises confidence questions about true superiority

P-Value Definition and Interpretation

  • P-values: numbers between 0 and 1
  • Quantify confidence that Drug A different from Drug B
  • Closer to 0 = more confidence in difference
  • Common threshold: 0.05
    • If no real difference exists, only 5% of experiments would yield wrong decision

P-Value Threshold Examples

  • Same drug given to two groups (Drug A to both)
    • Any differences 100% attributable to random factors
    • P-value = 0.9 (much larger than 0.05)
    • Conclusion: fail to see difference
  • Rare scenario: all allergic people in one group, strong placebo responders in other
    • P-value = 0.01
    • Would incorrectly conclude groups different (false positive)

False Positives and Threshold Selection

  • False positive: small p-value when no real difference exists
  • 0.05 threshold = 5% false positive rate
  • Stricter thresholds for critical decisions:
    • 0.00001 threshold = 1 false positive per 100,000 experiments
  • Relaxed thresholds for less critical decisions:
    • 0.2 threshold = 2 false positives per 10 experiments (ice cream truck example)
  • 0.05 most common - balance between accuracy and cost

Hypothesis Testing Terminology

  • Null hypothesis: drugs are the same
  • P-value helps decide whether to reject null hypothesis
  • Statistical decision making framework

P-Value Limitations - Effect Size

  • Small p-value ≠ large effect size
  • P-value indicates difference exists, not magnitude
  • Example 1: 6-point difference, p-value = 0.24 (not significant)
  • Example 2: 1-point difference with more people, p-value = 0.04 (significant)
  • Key insight: sample size affects p-value independent of effect size

Final Example Results

  • Experiment p-value = 0.24
  • Not confident Drug A different from Drug B (above 0.05 threshold)

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome