AI summary by TwinMind
Zapier’s Mike Knoop launches ARC Prize to Jumpstart New Ideas for AGI | Training Data
Sequoia Capital
55m 13s
Mike Knoop Background and Zapier's AI Integration
- Mike Knoop: co-founder of Zapier, recently co-founded Arc AGI Prize with François Chollet
- Zapier: workflow automation platform supporting 6,000+ integrations (Salesforce, Gmail, etc.)
- Designed for non-technical users (majority don't self-identify as programmers)
- Mission aligns with AI promise: software doing work for humans
- Mike's AI journey: curious since college, tracked progress, gave company presentation on GPT-3 paper
- January 2022 Chain of Thought paper was pivotal moment
- Surprised by "let's think step by step" technique
- Realized language models could be reasoning tools beyond completion engines
- Gave up executive team role to become AI researcher alongside co-founder Brian
Zapier's AI Implementation and Success Metrics
- Over 50% of company uses AI daily (measured via internal platform usage)
- Approaching 10 million AI tasks per month run rate
- Potentially largest automated AI platform globally for agentic systems
- Key success story: Zap templates automation achieving 100x labor enhancement
- Previously: contractors created 10 templates/day manually
- Now: 1,000 templates/day with AI system
- Human moved from "do loop" to "review loop"
- AI generates inspirational use cases + technical field mappings
- Bad outputs discarded due to low generation cost
- Backlog of 1 million templates being systematically addressed
- AI deployment strategy: target high-volume manual work near important business conversions
- Current AI adoption limited to low-trust/low-risk scenarios due to accuracy concerns
Arc AGI Definition and Motivation
- Traditional AGI definitions: undefinable vs. "system doing majority of economically useful work"
- François Chollet's definition: "efficiency of acquiring new skill"
- Key insight: current AI systems can't generalize to new domains
- Example: AI beating chess/go/poker can't be retrained for other games
- Requires complete architectural redesign for each new task
- Contrasts with human ability to learn new games in hours
- Arc AGI measures ability to rapidly gain skill for novel tasks
- Designed to resist memorization through completely novel tasks + private test set
- Unbeaten since 2019, showing deceleration in progress over time
- Current state-of-the-art: 39% (target: 85% for grand prize)
Arc AGI Competition Structure and Rules
- Hosted on Kaggle with specific constraints:
- 1 P100 GPU for 12 hours maximum
- No internet access (prevents using frontier closed models)
- Compute limit enforces efficiency requirement
- Prevents brute force approaches
- Forces researchers to develop generalizable solutions
- Mimics human problem-solving (testing 3-5 hypotheses vs. exhaustive search)
- Internet restriction reduces cheating/contamination risks
- Prize structure: $600K total, targeting 85% accuracy threshold
- Additional $150K public leaderboard for broader participation (allows frontier models)
Current Arc AGI Progress and Approaches
- Historical progression:
- 2019-2020: 20% baseline from small Kaggle competition
- 20-30%: hand-crafted domain-specific languages (DSL) + brute force search
- 30-40%: test-time fine-tuning with code generation models (Jack Cole's approach)
- Expected 50% achievable through ensembling existing techniques
- 85% solution likely requires: deep learning-guided DSL generator + program synthesis engine
- Current standout: Ryan Greenblatt's GPT-4o approach
- Outer loop around GPT-4o generating reasoning traces
- Tests patterns against demonstrations, applies working solutions
- Achieving ~40-41% accuracy
- Costs $10K for full evaluation (8,000 reasoning samples)
- Demonstrates novel frontier LLM reasoning technique
Research Community and Market Dynamics
- Most Arc competitors historically: outsiders to AI field
- Backgrounds: mechanical engineering, video game programming, physics
- Not big lab researchers
- Breakthrough likely from outsider with different perspective/cross-pollination
- Current AI field issues:
- Closed-source frontier research (GPT-4, Gemini papers lack technical details)
- Contrasts with open science that enabled transformer development chain
- Market dynamics pushing toward scale-only narrative
- Most labs internally working on new ideas despite public scale messaging
- Arc Prize goals: counterbalance closed research, accelerate open progress
- Low awareness of Arc despite François Chollet's high profile
- Benchmarks gain popularity through progress, Arc's slow progress limited attention
Technical Architecture and Future Directions
- Language models: high-dimensional memorization systems
- Effective for adjacent contexts but limited novel task generalization
- Current limits: can't compose training data for unseen tasks
- Transformers potentially capable of 100% accurate deductive reasoning
- Challenge: backpropagation ineffective for teaching deductive reasoning
- Need better learning algorithms/outer loops around transformers
- Deep learning likely part of solution, but not pure deterministic programming
- Rich search space of alternative architectures (9-10 mainline architectures)
- LLMs probably subcomponent rather than entire solution architecture
Implications and Future Vision
- Arc solution provides exacting accuracy + generalization from small priors
- Addresses #1 application layer problem: hallucination/consistency issues
- Enables higher-trust AI deployment in moderate/high-risk scenarios
- AGI advent likely gradual/incremental vs. sudden moment
- Allows society time to adapt, make decisions about deployment
- Empirical evidence-based policy vs. theoretical doom scenarios
- Risk of premature regulation cutting off beneficial futures
- Need continued new ideas from diverse researchers/students/labs
- Solution must be open-sourced (prize requirement) to accelerate progress
Competition Status and Predictions
- 2 weeks post-launch: phenomenal response
- Trending on Twitter twice, #1 Kaggle competition globally
- Over 1M social media views across channels
- Two competitor camps emerging:
- Mission-driven: agree new ideas needed
- Skeptical: trying to prove LLMs/scale sufficient
- Timeline predictions:
- Pre-launch expectation: 3-4 years to 85%
- Current: likely 50% during competition period (ends mid-November 2024)
- 85% grand prize: would be surprised but hopeful
- Why now factors:
- More compute/engineering experience with deep learning systems
- Lower costs make previously impractical approaches feasible
- Increased awareness through prize attention
- Risk: may actually not be "why now" - could need more fundamental breakthroughs
Personal Motivations and Broader Impact
- Desire to "live in the future" - pull forward beneficial technological progress
- Current narrow AI always rate-limited by humans in the loop
- AGI enables AI systems to invent/discover/innovate alongside humans
- Accelerate progress in physics, pharmaceuticals, universe understanding
- Remove human bottleneck from frontier advancement
- Optimistic about gradual rollout with empirical feedback loops
- Emphasis on open progress vs. closed development for societal benefit
