AI summary by TwinMind

AI Slop Is Spreading In Science, Too

Sabine Hossenfelder

7m 50s

AI Impact on Scientific Research Quality

  • Years of concern about AI impact on science amid existing problems
    • Research areas already struggling with nonsense research
    • Peer review system failing
    • Situation deteriorating
  • Examples of suspected AI-generated content:
    • Trump administration tariff formula
    • UK tech secretary using ChatGPT for policy advice
    • Terrence Howard's "solution" to three-body problem (actor believes 1×1=2)
    • Half dozen "theories of everything" admittedly written with AI help

Mass Paper Generation Demonstration

  • January 2025: Economists demonstrated AI's paper production capability
    • Used GPT and Claude to analyze stock data
    • Generated 288 complete finance papers
    • Not gibberish - used reasonable protocol analyzing real data
  • Potential for replication in other fields:
    • Cosmology, particle physics possible targets
    • Could analyze existing datasets for dark matter models
    • Thousands of papers generatable instantly
    • Economic incentive driving adoption

Current AI Detection Challenges

  • Early AI papers trackable by phrases like "as of my last knowledge update"
  • January research: AI-generated papers most common in fish research
  • Advanced generative AI eliminates obvious text markers
  • ProofFake company findings (January):
    • AI-generated scientific images now extremely sophisticated
    • Most researchers cannot distinguish AI images from real ones
    • Major problem for biology and materials science fields

Case Study: "Vegetative Electron Microscopy"

  • Nonsensical term appearing in 20+ scientific papers
  • Origin: 1959 two-column article misreading
    • "Vegetative cell wall" in one column
    • "Electron microscopy" in other column
  • AI propagation aided by Farsi language similarity (scanning ≈ vegetative)
  • Now published in major publisher journals
  • Represents detectable cases - vast majority likely undetected

Scientific Misconduct Detection Problems

  • PubPeer platform for discussing paper problems
  • System vulnerable to abuse (fake reviews equivalent)
  • Sylvian Bernes case study:
    • Email requesting paper retractions: "I don't like this group of researchers"
    • Threat: "I'll be happy if first author loses his job"
    • Follow-up threat to post papers on PubPeer for retractions
  • Weaponization similar to fake restaurant reviews

Citation Manipulation and Fake Collaborations

  • Previous case: Cat with academic citations from fake papers
    • Uploaded to ResearchGate
    • Indexed by Google Scholar
    • "Possibly tenured by now"
  • March preprint: Cluster of authors boosting citations via same method
  • ResearchGate recommended Geoffrey Hinton check paper he "wrote" with Yann LeCun
    • News to both supposed authors
    • Fake collaboration detection

Future Implications

  • Science approaching point of drowning in "AI slop"
  • Potential positive outcome: Force addressing publish-or-perish culture
  • Current trajectory: Science focused on generating "statistically plausible sentences" rather than finding truth
  • AI represents beginning of new civilization phase

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome