AI summary by TwinMind

The method that can "prove" almost anything - James A. Smith

TED-Ed

5m 5s

P-Value Methodology and Tea Experiment

  • 2011 researchers conducted study showing listening to songs makes people younger
    • Used real people, truthful data, standard statistical analyses
    • Study designed to demonstrate statistical method misuse
  • Tea experiment explanation of p-value concept
    • 8 cups: 4 milk-first, 4 tea-first
    • Participant must identify which is which by taste
    • 70 different ways to sort into two groups of four
    • Only 1 correct arrangement
  • Null hypothesis definition: participant can't distinguish teas
    • Still gets right answer 1 in 70 times by chance
    • 1/70 = roughly 0.014 (p-value)
    • P-value ≤0.05 considered statistically significant in many fields
    • P-value of 0.014 would reject null hypothesis

P-Value Limitations and Confusion

  • P-values commonly used by researchers and journals but confusing even for scientists
  • P-value only tells probability of getting result assuming null hypothesis true
    • If participant correctly sorts teas, p-value = probability of doing so assuming she can't tell difference
    • Reverse not true: p-value doesn't tell probability she can taste difference
  • Scientific community uses p-values because they give reliable indication when used correctly
    • Don't directly state probability results due to random chance
    • Usually reliable indicator if applied properly

P-Hacking Problem

  • Real studies more complex than tea experiment
  • Scientists can test research questions multiple ways
    • Some tests produce statistically significant results, others don't
    • Each additional test increases chance of false positive
  • P-hacking definition: searching for low p-value, presenting only that analysis
    • Like throwing darts until hitting bullseye, then claiming only threw successful dart

Music Study P-Hacking Example

  • Researchers played different songs to three participant groups
  • Collected extensive information about participants
  • Published analysis included only 2 out of 3 groups
  • Used only participants' fathers' age from all collected data
    • Justified as "control for variation in baseline age across participants"
  • Paused experiment after every 10 participants
    • Continued if p-value >0.05
    • Stopped when p-value <0.05
  • Final result: participants hearing one song were 1.5 years younger than other group
    • P-value = 0.04

Solutions and Meta-Science

  • P-hacking usually harder to spot because results aren't obviously impossible
  • Pre-registration solution: detailed experiment and analysis plan beforehand
    • Others can check the plan
    • Prevents researchers from trying different analyses until finding significant result
  • Meta-science field emerging: science studying scientific practices to improve them

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome