AGI Threshold via ARC Benchmark Performance

• Multimodal model achieving 80% on ARC (average human performance) could indicate AGI arrival within a year
• Preference for LLM solving ARC at 80% without explicit training on ARC-anticipating information
• ARC designed as novel intelligence test - each problem requires new type of reasoning
• ARC released 4+ years ago, has resisted memorization attempts, passed test of time
• Not perfect benchmark - vulnerable to brute force approaches

Memorization vs Intelligence Debate

• Concern: Creating hundreds of thousands of ARC tasks by hand + programmatic variations
  • Could generate hundreds of millions of tasks
  • Brute forcing task space creates overlap between training/test sets
  • Enables high scores through scale rather than intelligence
• Counter-argument: Human learning also involves memorization
  • Children learning algebra/calculus aren't "memorizing" - they're learning
  • Humans memorize algorithms/programs then reapply them
  • Addition example: Kids don't derive from set theory, they memorize technique
  • School learning is mostly memorization of skills/techniques

Intelligence as Pathfinding Algorithm Metaphor

• Intelligence = pathfinding algorithm in future situation space
• Analogy to RTS game development with 2D maps
• Partial information constraints:
  • Fog of war (unexplored areas)
  • Historical knowledge of explored areas (not current state)
• Goal-setting enables optimal path determination
• Perfect information scenario would allow pure memorization of all paths
• Real life prevents this due to unpredictable future changes

Current AI Model Limitations and Capabilities

• Current models vastly underparameterized vs human brain (flops/parameters)
• Won't generate new theorems like smartest humans
• Most humans can't do theorem generation either
• Most human capability resembles "memorization" - learned skills/techniques

Economic Automation Potential

• Scenario: All remote workers replaced by AI
• Method: Record all remote worker screens, understand performed skills
• Create synthetic training data from recorded activities
• Train models to replicate all remote work functions
• Result: Trillions of dollars economic activity from AI remote workers
• Still operates in memorization regime for static distributions