AI summary by TwinMind

AIs Predict Research Results Without Doing Research

Sabine Hossenfelder

7m 4s

Study Overview and Methodology

  • Remarkable study testing AI ability to predict scientific results from existing literature
  • Large language models fed existing scientific literature predicted study results with amazing accuracy
  • Models predicted results of studies they were not trained on
  • Trained several LLMs including GPT-4 and Meta's LLaMA 2 on 120,000+ full text neuroscience articles (2002-2022)
  • Recruited ~170 neuroscience experts for comparison testing

AI vs Human Performance Results

  • Presented both AI and humans with original vs altered neuroscience abstracts
  • Asked to identify correct version
  • General purpose LLMs (GPT-4): 81% accuracy
  • Human experts: 63% accuracy average
  • Fine-tuned model: nearly 85% accuracy
  • AI significantly outperformed human experts

Historical Context - Dan Swanson's Work (1980s)

  • Computer scientist studied reference distribution patterns in scientific papers
  • Found separate research topics with few cross-references but common third topic references
  • Indicated hidden links between separate research areas
  • Concrete example discovered:
    • Fish oils benefit blood vessels (known)
    • Raynaud's syndrome patients improve with better circulation (known)
    • Predicted fish oil would help Raynaud's syndrome patients
    • 1993 clinical trial confirmed hypothesis was correct
  • Scientific community missed obvious connection despite evidence being available

Material Science Neural Network Analysis

  • California researchers analyzed material science literature with simple neural network
  • Used word analysis of paper abstracts (not citations like Swanson)
  • Found thermoelectric materials associated with word "thermoelectric" before humans knew they were thermoelectric
  • Knowledge existed in word relations before human discovery
  • Also predicted new thermoelectric materials from existing literature

AI Advantages Over Human Scientists

  • Can read scientific literature much faster than humans
  • Scientific publications increasing exponentially
  • Too fast for any human to follow
  • Entire physics literature doubled since speaker's PhD
  • Rapid growth forces scientists into specialized niches
  • Specialization unhelpful for knowledge discovery
  • Creates missed connections waiting to be found

Applications and Future Implications

  • Analysis useful for identifying most promising new studies
  • Can identify key questions likely to unlock further progress
  • Humans can't digest all literature or draw relevant connections
  • Funding agencies should use AI-powered analysis to identify research to fund
  • Speaker's 10-year-old proposal to European Research Council was declined
    • Declined because speaker lacked data analysis expertise
    • Funding was needed to hire expert analysts
    • Without funding, analysis wasn't done
  • Soon AI will perform this analysis independently

Educational Content Promotion

  • Brilliant.org sponsorship segment
  • Interactive visualizations and follow-up questions
  • Covers science, computer science, mathematics topics
  • Speaker has quantum mechanics course on platform
  • 30-day free trial and 20% off annual subscription offer

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome