AI summary by TwinMind

John Schulman (OpenAI Cofounder) - Reasoning, RLHF, & Plan for 2027 AGI

Dwarkesh Patel

1h 36m 54s

Pre-Training vs Post-Training Fundamentals

  • Pre-training: models learn to imitate all internet content (websites, code)
    • Trained to maximize likelihood, predict next token
    • Creates calibrated models that can generate diverse content and assign probabilities
    • Base model can take on different personas, generate various content types
  • Post-training: targets narrower behavior range
    • Shapes model to behave like chat assistant with specific persona
    • Optimizes for outputs humans find useful vs raw web content imitation
    • Creates helpful, question-answering behavior rather than person imitation

Future AI Capabilities Timeline (1-5 Years)

  • 1-2 years: models capable of whole coding projects
    • Move from single function suggestions to multi-file projects
    • High-level instructions → model writes/tests/iterates multiple files
    • Fundamental unlock: acting coherently for extended periods
  • Capability improvements through:
    • Training on harder, longer-horizon tasks (currently mostly single-step data)
    • RL training for multi-step projects
    • Better error recovery and edge case handling
    • More sample-efficient learning from limited data

Long-Horizon RL and Scaling Challenges

  • Current models act coherently ~5 minutes, need hour/week/month-long tasks
  • Scaling may not follow clean 10x compute law per benchmark
  • Phase transitions possible where same mechanisms work across time scales
  • Humans use similar mental machinery for planning month vs year vs decade ahead
  • Language enables describing all time scales, planning progress toward distant goals

Potential Bottlenecks Beyond Long-Horizon Capability

  • Human expert experience: taste, ambiguity handling
  • Research-specific considerations beyond pure task execution
  • Physical world affordances and UI interactions
  • Mundane barriers (website compatibility, multimodal interfaces)
  • Models will likely use human-designed UIs initially
  • Some websites may optimize for AI interaction (better text representations)

Generalization Evidence in Post-Training

  • English fine-tuning automatically improves other languages
  • Text-only fine-tuning improves multimodal behavior
  • 30 examples of "model limitations" generalized to many uncovered capabilities
  • Small amounts of targeted data can fix broad behavioral issues
  • Models show strong transfer between related domains

AGI Timeline and Safety Considerations

  • Schulman: reasonable to plan for AGI possibility soon, though not expecting next year
  • If AGI arrives unexpectedly early:
    • Slow down training/deployment until safety understanding improves
    • Careful sandboxing of smarter systems
    • Coordination needed among major AI entities to avoid race dynamics
  • Preferred scenario: gradual capability increases with corresponding safety improvements
  • Avoid discontinuous jumps that create coordination challenges

Safety and Alignment Approaches

  • Defense in depth strategy:
    • Well-behaved models with strong moral compass
    • Resistance to takeover/severe misuse
    • Comprehensive monitoring systems
    • Simulated deployment testing (red teaming)
  • Current RLHF feels safe: models only optimize for human-approved text
  • Long-horizon RL introduces new risks: models taking unusual actions during task execution
  • Instrumental convergence concerns for poorly specified tasks

Current RLHF System Characteristics

  • Functions like drive/goal in human psychology
  • Models trained to maximize human approval via reward model
  • Creates consistent "helpful assistant" persona
  • Results in verbose, bullet-point heavy, formal responses
  • May be due to: training on single messages, labeler preferences, speed considerations
  • Unintentional distillation between providers (contractors using other chatbots)

Reasoning and Test-Time Computation

  • Two approaches for better reasoning:
    • Training-time: learn to follow correct reasoning chains
    • Test-time: use compute during inference for step-by-step reasoning
  • Best results likely combine both approaches
  • Reasoning requires test-time computation by definition
  • Models need practice (training-time computation) to improve

Memory and Learning Systems

  • Gap between pre-training (trillions of tokens) and in-context learning
  • Need for medium-term memory: more than context, less than pre-training scale
  • Active learning: models understanding knowledge gaps, seeking relevant information
  • Introspection capabilities could enable self-directed learning
  • Long context windows (100K-1M tokens) reducing need for alternative approaches
  • Online learning + cognitive introspection skills likely complementary

Research and Development Insights

  • GPT-4 ELO score improved 100 points since release, mostly from post-training
  • Multiple improvement axes: data quality, quantity, process iterations
  • Post-training creates significant competitive moat due to complexity
  • Requires skilled teams, tacit knowledge, organizational expertise
  • ML field generally healthy compared to social sciences due to practical grounding

ChatGPT Development History

  • Built on GPT-3.5 (finished training early 2022)
  • Evolved from instruction-following models and web browsing research
  • Chat format more intuitive than instruction completion for users and labelers
  • Required iterative supervised fine-tuning, not just one-shot human data training
  • Mixed instruction and chat datasets for best-of-both-worlds approach

Technical Scaling and Efficiency

  • Bigger models more sample-efficient: learn same capability with less data
  • Possible explanations: ensemble of parallel circuits, mixture of experts concept
  • Bigger models have larger "library" of computational circuits
  • No clear understanding of why parameter scaling works so well
  • Continued efficiency improvements in both pre-training and post-training

Data and Training Considerations

  • Approaching but not immediately hitting data wall
  • Pre-training nature will change as high-quality data becomes scarce
  • Post-training compute allocation likely to increase significantly
  • Model-generated content higher quality than much web content
  • Synthetic data and model self-improvement becoming more important

Deployment and Integration

  • AI still small fraction of economy, many jobs can't be helped yet
  • Growth from both capability improvements and better integration
  • Expect widespread adoption across more sophisticated tasks
  • Science acceleration through literature comprehension and data analysis
  • Form factor evolution: from search engine to collaborative project partner
  • Proactive suggestions and background work rather than one-off queries

Stakeholder Alignment Framework

  • Four stakeholder categories: end user, developer, platform (OpenAI), humanity
  • Model spec addresses conflicting stakeholder demands
  • Generally follow user instructions while preventing harm to others
  • Avoid paternalistic behavior, maintain neutrality on opinions
  • Balance helpfulness with harm prevention

Future Concerns and Considerations

  • Economic pressures may force removal of human oversight
  • Amdahl's law: human bottlenecks limit firm efficiency gains
  • Regulation challenges across companies and countries
  • Need coordination among model providers or universal regulatory regimes
  • AI-run firms may have higher tail risks despite better average performance

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome