AI summary by TwinMind
The Reproducibility Crisis
Sabine Hossenfelder
21m 32s
Reproducibility Crisis Overview
- Psychology facing major reproducibility problem over past 10-15 years
- Big study found only 30-40% of published psychology results reproduce
- Problem extends beyond psychology to biomedical sciences
- Pharmaceutical industry concerned - can't reproduce basic experimental results from academic labs
- Academy of Medical Sciences symposium brought together diverse stakeholders including physicists, industry representatives
Four Horsemen of Reproducibility Apocalypse
- Publication Bias: Much easier to publish positive/exciting results than null results
- Creates "file drawer" effect - unpublished negative results distort literature
- Most scientists won't try publishing non-significant results
- Affects cumulative research process across disciplines
- P-hacking: Selecting specific results within study to focus on
- Analyzing data multiple ways, reporting only significant results
- Collecting data on many variables, reporting only exciting ones
- Many psychologists don't realize this distorts literature
- Think p-value <0.05 automatically means something important
- Low Statistical Power: Studies with samples too small to detect effects
- Problem discussed since 1970s but often dismissed as statistician pedantry
- Effect sizes in psychology typically quite small, requiring much larger samples
- Sample size varies by subarea - special populations may take 3 years to collect 30 children
- Brain imaging expensive (~£500 per scan) limiting sample sizes
- Solution: larger collaborative enterprises
- HARKing (Hypothesizing After Results Known): Looking at data first, then creating hypothesis
- Using same data to generate AND test hypothesis
- Researcher tells "good story" but violates proper scientific method
- Ties into p-hacking problem
Solutions and Reforms
- Registered Reports: New publication model addressing all four problems
- Submit introduction/methods before data collection
- Highly specified methods with analysis scripts, simulated data
- Reviewers evaluate question interest, methodology, adequate power
- In-principle acceptance before data collection
- Final publication based on following pre-registered plan, not results
- Developed by Chris Chambers (University of Cardiff)
- Started 6-7 years ago with one journal, now 200-300 journals offer it
- Expanding beyond psychology to life sciences, sociology, economics
- Educational Approaches: Using simulated data where researcher knows "ground truth"
- Demonstrate how to find significant results when nothing there
- Show how small samples miss real effects
- Not yet routine in statistics education
Historical Context and Social Change
- Statistical problems known since 1970s but not taken seriously
- Two factors enabled recent change:
- Genuine misunderstanding of severity
- Social media giving voice to junior researchers
- Early career scientists becoming "militant" about reform
- Previously could only write letters to editors; now can use Twitter for immediate criticism
Institutional Reform Efforts
- Oxford running "Reproducible Research Oxford" course for early career researchers (4 years)
- Journal club "Reproducibility Tea"
- University-wide bid led by Lera Fortunato (Anthropology)
- Cross-disciplinary approach including humanities (museum collections documentation)
- Focus on credible science vs flashy journal publications
Funding and Incentive Changes
- Wellcome Trust, MRC, BBSRC supporting reform
- Funders don't want to support non-credible research
- New grant proposal requirements emerging
- Universities changing hiring/promotion criteria
- International leadership - Oxford only place doing all-discipline convergence
- Initiative launched January (one month before interview)
Public Communication Challenges
- Debate over publicizing problems vs maintaining public confidence
- Risk of "weaponization" by those opposing regulations
- US Trump administration using reproducibility concerns to dismiss environmental protections
- Asbestos data from pre-open-data era being dismissed
- Climate change science being questioned
- Balance needed: openness about problems while preventing misuse
- Best approach: make science genuinely better and faster at self-correction
Current State and Examples
- Bishop increasingly skeptical as reviewer when data not open
- Recent paper review showed clear p-hacking, authors claimed "everybody does it"
- Classic psychology guidebook explicitly recommends p-hacking and HARKing
- Many "robust" phenomena in psychology textbooks now questionable
- Need to be "shamefaced" about past problems while working on solutions
