Intelligence Explosion Model Discussion

• Replace AI researchers with automated AI researchers to speed up progress
• Question: Would 1000 AI researchers create intelligence explosion?
• Current bottlenecks: compute to run experiments + "taste" for right research directions
  • Less bound by engineering work than by compute and inference judgment
  • Making difficult inferences on imperfect information is key constraint

Compute Scaling Elasticity

• 10x H100s would make Gemini program ~5x faster
• Elasticity coefficient: 0.5 (significant responsiveness to compute increases)
• More compute directly converts to research progress
• Fixed compute allocation split between inference and experiments

Strategic Compute Allocation Decisions

• Pre-training teams must decide compute allocation between research vs scaling
• Need to train big models to understand emergent properties at scale
• Risk of going off successful scaling path if only doing small-scale research
• Must constantly invest in frontier-scale runs alongside research

AI-Accelerated Research Scenarios

• Two pathways for AI speeding up research:
  1. AI meaningfully speeds up algorithmic progress (requires more compute)
  2. AI output becomes crucial ingredient (synthetic data generation)
• AI as "fantastic co-pilot" - easier to onboard than humans
• Super long context, smart models for immediate deployment on subtasks

Research Workflow and Bottlenecks

• Core cycle: idea generation → proving at different scales → interpretation of failures
• Most surprising bottleneck: interpreting and understanding what goes wrong
• Not about generating thousands of lines of code
• Many researchers have long lists of ideas - filtering/prioritization is hard
• "Shot calling under imperfect information" for idea selection

Imperfect Information Problem

• Scaling law predictions (like GPT-4 paper dots/curves) don't always hold
• Trends that help at smaller scales can hurt at larger scales
• Making guesses based on trend lines and intuitive feelings
• Published smooth curves hide "graveyard" of failed first runs

Research Execution Challenges

• Constant experimentation and intuiting what went wrong
• Interpretability team: many ideas, need engineering skill for rapid iteration
• "Engineering in quotes" - much is research for quick experiment cycles
• Ruthless prioritization separates quality research from unsuccessful attempts

Effective Researcher Characteristics

• Don't get attached to familiar solution toolboxes
• Attack problems directly with expanded toolsets
• Draw from reinforcement learning, optimization theory, systems understanding
• Fast iteration ability - cycle time separates best researchers
• Machine learning research is highly empirical
• Best researchers can try experiments "really really really really really fast"

Evolutionary Optimization Perspective

• AI community doing "greedy evolutionary optimization" over possible architectures
• Solutions might end up more brain-like than expected due to empirical approach