Introduction to P-Values

• Josh Starmer introduces p-values concept using drug comparison example
• Focus on what p-values are and how to interpret them

Drug Testing Scenario Setup

• Comparing Drug A vs Drug B effectiveness
• Initial test: 1 person each drug
  • Drug A: 1 person cured
  • Drug B: 1 person not cured
• Cannot conclude Drug A better - too many confounding factors:
  • Bad medication interactions
  • Rare allergies
  • Missed doses
  • Placebo effects

Expanding Sample Size

• Second test: 2 people per drug
  • Drug A: both people cured
  • Drug B: 1 cured, 1 not cured
• Still inconclusive due to potential random factors
  • Mislabeled drugs
  • Individual variations

Large Scale Results - Clear Difference

• Drug A results: 1043 cured, 3 not cured (99.7% cure rate, 1046 total)
• Drug B results: 2 cured, 1432 not cured (0.1% cure rate, 1434 total)
• Clear superiority of Drug A
• Too many people cured by Drug A, too few by Drug B for random chance explanation

Ambiguous Results Scenario

• Drug A: 37% cure rate
• Drug B: 31% cure rate
• Smaller difference raises confidence questions about true superiority

P-Value Definition and Interpretation

• P-values: numbers between 0 and 1
• Quantify confidence that Drug A different from Drug B
• Closer to 0 = more confidence in difference
• Common threshold: 0.05
  • If no real difference exists, only 5% of experiments would yield wrong decision

P-Value Threshold Examples

• Same drug given to two groups (Drug A to both)
  • Any differences 100% attributable to random factors
  • P-value = 0.9 (much larger than 0.05)
  • Conclusion: fail to see difference
• Rare scenario: all allergic people in one group, strong placebo responders in other
  • P-value = 0.01
  • Would incorrectly conclude groups different (false positive)

False Positives and Threshold Selection

• False positive: small p-value when no real difference exists
• 0.05 threshold = 5% false positive rate
• Stricter thresholds for critical decisions:
  • 0.00001 threshold = 1 false positive per 100,000 experiments
• Relaxed thresholds for less critical decisions:
  • 0.2 threshold = 2 false positives per 10 experiments (ice cream truck example)
• 0.05 most common - balance between accuracy and cost

Hypothesis Testing Terminology

• Null hypothesis: drugs are the same
• P-value helps decide whether to reject null hypothesis
• Statistical decision making framework

P-Value Limitations - Effect Size

• Small p-value ≠ large effect size
• P-value indicates difference exists, not magnitude
• Example 1: 6-point difference, p-value = 0.24 (not significant)
• Example 2: 1-point difference with more people, p-value = 0.04 (significant)
• Key insight: sample size affects p-value independent of effect size

Final Example Results

• Experiment p-value = 0.24
• Not confident Drug A different from Drug B (above 0.05 threshold)