Study Overview and Methodology

• Remarkable study testing AI ability to predict scientific results from existing literature
• Large language models fed existing scientific literature predicted study results with amazing accuracy
• Models predicted results of studies they were not trained on
• Trained several LLMs including GPT-4 and Meta's LLaMA 2 on 120,000+ full text neuroscience articles (2002-2022)
• Recruited ~170 neuroscience experts for comparison testing

AI vs Human Performance Results

• Presented both AI and humans with original vs altered neuroscience abstracts
• Asked to identify correct version
• General purpose LLMs (GPT-4): 81% accuracy
• Human experts: 63% accuracy average
• Fine-tuned model: nearly 85% accuracy
• AI significantly outperformed human experts

Historical Context - Dan Swanson's Work (1980s)

• Computer scientist studied reference distribution patterns in scientific papers
• Found separate research topics with few cross-references but common third topic references
• Indicated hidden links between separate research areas
• Concrete example discovered:
  • Fish oils benefit blood vessels (known)
  • Raynaud's syndrome patients improve with better circulation (known)
  • Predicted fish oil would help Raynaud's syndrome patients
  • 1993 clinical trial confirmed hypothesis was correct
• Scientific community missed obvious connection despite evidence being available

Material Science Neural Network Analysis

• California researchers analyzed material science literature with simple neural network
• Used word analysis of paper abstracts (not citations like Swanson)
• Found thermoelectric materials associated with word "thermoelectric" before humans knew they were thermoelectric
• Knowledge existed in word relations before human discovery
• Also predicted new thermoelectric materials from existing literature

AI Advantages Over Human Scientists

• Can read scientific literature much faster than humans
• Scientific publications increasing exponentially
• Too fast for any human to follow
• Entire physics literature doubled since speaker's PhD
• Rapid growth forces scientists into specialized niches
• Specialization unhelpful for knowledge discovery
• Creates missed connections waiting to be found

Applications and Future Implications

• Analysis useful for identifying most promising new studies
• Can identify key questions likely to unlock further progress
• Humans can't digest all literature or draw relevant connections
• Funding agencies should use AI-powered analysis to identify research to fund
• Speaker's 10-year-old proposal to European Research Council was declined
  • Declined because speaker lacked data analysis expertise
  • Funding was needed to hire expert analysts
  • Without funding, analysis wasn't done
• Soon AI will perform this analysis independently

Educational Content Promotion

• Brilliant.org sponsorship segment
• Interactive visualizations and follow-up questions
• Covers science, computer science, mathematics topics
• Speaker has quantum mechanics course on platform
• 30-day free trial and 20% off annual subscription offer