AI summary by TwinMind
The method that can "prove" almost anything - James A. Smith
TED-Ed
5m 5s
P-Value Methodology and Tea Experiment
- 2011 researchers conducted study showing listening to songs makes people younger
- Used real people, truthful data, standard statistical analyses
- Study designed to demonstrate statistical method misuse
- Tea experiment explanation of p-value concept
- 8 cups: 4 milk-first, 4 tea-first
- Participant must identify which is which by taste
- 70 different ways to sort into two groups of four
- Only 1 correct arrangement
- Null hypothesis definition: participant can't distinguish teas
- Still gets right answer 1 in 70 times by chance
- 1/70 = roughly 0.014 (p-value)
- P-value ≤0.05 considered statistically significant in many fields
- P-value of 0.014 would reject null hypothesis
P-Value Limitations and Confusion
- P-values commonly used by researchers and journals but confusing even for scientists
- P-value only tells probability of getting result assuming null hypothesis true
- If participant correctly sorts teas, p-value = probability of doing so assuming she can't tell difference
- Reverse not true: p-value doesn't tell probability she can taste difference
- Scientific community uses p-values because they give reliable indication when used correctly
- Don't directly state probability results due to random chance
- Usually reliable indicator if applied properly
P-Hacking Problem
- Real studies more complex than tea experiment
- Scientists can test research questions multiple ways
- Some tests produce statistically significant results, others don't
- Each additional test increases chance of false positive
- P-hacking definition: searching for low p-value, presenting only that analysis
- Like throwing darts until hitting bullseye, then claiming only threw successful dart
Music Study P-Hacking Example
- Researchers played different songs to three participant groups
- Collected extensive information about participants
- Published analysis included only 2 out of 3 groups
- Used only participants' fathers' age from all collected data
- Justified as "control for variation in baseline age across participants"
- Paused experiment after every 10 participants
- Continued if p-value >0.05
- Stopped when p-value <0.05
- Final result: participants hearing one song were 1.5 years younger than other group
- P-value = 0.04
Solutions and Meta-Science
- P-hacking usually harder to spot because results aren't obviously impossible
- Pre-registration solution: detailed experiment and analysis plan beforehand
- Others can check the plan
- Prevents researchers from trying different analyses until finding significant result
- Meta-science field emerging: science studying scientific practices to improve them
