AI summary by TwinMind

The Reproducibility Crisis

Sabine Hossenfelder

21m 32s

Reproducibility Crisis Overview

  • Psychology facing major reproducibility problem over past 10-15 years
  • Big study found only 30-40% of published psychology results reproduce
  • Problem extends beyond psychology to biomedical sciences
  • Pharmaceutical industry concerned - can't reproduce basic experimental results from academic labs
  • Academy of Medical Sciences symposium brought together diverse stakeholders including physicists, industry representatives

Four Horsemen of Reproducibility Apocalypse

  • Publication Bias: Much easier to publish positive/exciting results than null results
    • Creates "file drawer" effect - unpublished negative results distort literature
    • Most scientists won't try publishing non-significant results
    • Affects cumulative research process across disciplines
  • P-hacking: Selecting specific results within study to focus on
    • Analyzing data multiple ways, reporting only significant results
    • Collecting data on many variables, reporting only exciting ones
    • Many psychologists don't realize this distorts literature
    • Think p-value <0.05 automatically means something important
  • Low Statistical Power: Studies with samples too small to detect effects
    • Problem discussed since 1970s but often dismissed as statistician pedantry
    • Effect sizes in psychology typically quite small, requiring much larger samples
    • Sample size varies by subarea - special populations may take 3 years to collect 30 children
    • Brain imaging expensive (~£500 per scan) limiting sample sizes
    • Solution: larger collaborative enterprises
  • HARKing (Hypothesizing After Results Known): Looking at data first, then creating hypothesis
    • Using same data to generate AND test hypothesis
    • Researcher tells "good story" but violates proper scientific method
    • Ties into p-hacking problem

Solutions and Reforms

  • Registered Reports: New publication model addressing all four problems
    • Submit introduction/methods before data collection
    • Highly specified methods with analysis scripts, simulated data
    • Reviewers evaluate question interest, methodology, adequate power
    • In-principle acceptance before data collection
    • Final publication based on following pre-registered plan, not results
    • Developed by Chris Chambers (University of Cardiff)
    • Started 6-7 years ago with one journal, now 200-300 journals offer it
    • Expanding beyond psychology to life sciences, sociology, economics
  • Educational Approaches: Using simulated data where researcher knows "ground truth"
    • Demonstrate how to find significant results when nothing there
    • Show how small samples miss real effects
    • Not yet routine in statistics education

Historical Context and Social Change

  • Statistical problems known since 1970s but not taken seriously
  • Two factors enabled recent change:
    • Genuine misunderstanding of severity
    • Social media giving voice to junior researchers
  • Early career scientists becoming "militant" about reform
  • Previously could only write letters to editors; now can use Twitter for immediate criticism

Institutional Reform Efforts

  • Oxford running "Reproducible Research Oxford" course for early career researchers (4 years)
  • Journal club "Reproducibility Tea"
  • University-wide bid led by Lera Fortunato (Anthropology)
  • Cross-disciplinary approach including humanities (museum collections documentation)
  • Focus on credible science vs flashy journal publications

Funding and Incentive Changes

  • Wellcome Trust, MRC, BBSRC supporting reform
  • Funders don't want to support non-credible research
  • New grant proposal requirements emerging
  • Universities changing hiring/promotion criteria
  • International leadership - Oxford only place doing all-discipline convergence
  • Initiative launched January (one month before interview)

Public Communication Challenges

  • Debate over publicizing problems vs maintaining public confidence
  • Risk of "weaponization" by those opposing regulations
  • US Trump administration using reproducibility concerns to dismiss environmental protections
  • Asbestos data from pre-open-data era being dismissed
  • Climate change science being questioned
  • Balance needed: openness about problems while preventing misuse
  • Best approach: make science genuinely better and faster at self-correction

Current State and Examples

  • Bishop increasingly skeptical as reviewer when data not open
  • Recent paper review showed clear p-hacking, authors claimed "everybody does it"
  • Classic psychology guidebook explicitly recommends p-hacking and HARKing
  • Many "robust" phenomena in psychology textbooks now questionable
  • Need to be "shamefaced" about past problems while working on solutions

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome