AI summary by TwinMind

Sholto Douglas & Trenton Bricken - How LLMs Actually Think

Dwarkesh Patel

3h 13m 13s

Context Windows and Their Importance

  • Long context capabilities dramatically underappreciated despite allowing million+ token inputs
  • Solves "onboarding problem" for models - can instantly understand entire codebases
  • Shows step-up in intelligence comparable to huge model scale increases
  • Models learn languages in-context better than human experts over months
  • Example: esoteric human language not in training data, model learns from context alone
  • Creates superhuman capability in information integration (humans can't hold million tokens in context)

In-Context Learning Mechanisms

  • In-context learning similar to gradient descent in attention operations
  • Model performs meta-learning - learning how to learn from context examples
  • Shows exact correspondence between gradient descent steps and attention layers
  • Enables dramatic sample efficiency improvements with scale
  • Training on long context tasks forces models to develop meta-learning abilities
  • Creates flexible, adaptive intelligence through better context utilization

Long Horizon Tasks vs Reliability

  • AI agents haven't succeeded due to reliability issues, not long context limitations
  • Need "nines of reliability" - small probability failures compound over long task chains
  • 99% → 90% success rate makes multi-step tasks much less likely to succeed
  • Academic evals focus on single problems, not chained multi-hour/multi-day tasks
  • Sweet Bench represents progress toward realistic long-horizon evaluation
  • Economic impact requires understanding success rates over different time horizons

Attention Mechanisms and Computational Costs

  • Quadratic attention cost often dominated by MLP blocks in practice
  • At inference, attention is linear (one query against many KV vectors)
  • Actual bottlenecks differ from theoretical scaling concerns
  • Google achieving million-token windows suggests quadratic costs manageable
  • Research community overemphasized attention scaling problems

Forward Pass vs Backward Pass Learning

  • Trend toward more learning happening in forward pass (in-context) vs backward pass (training)
  • Represents progression from bottom-up evolutionary learning to top-down reasoning
  • Chain of Thought as adaptive compute - more forward passes for harder problems
  • Models can transmit information through KV cache beyond just output tokens
  • Steganography possible in attention mechanisms during reasoning

Brain-AI Analogies and Attention

  • Cerebellum circuit matches attention operation almost exactly
  • Associative memory algorithm from 1980s identical to cerebellar architecture
  • Same architecture appears across evolution (fruit flies to humans)
  • Three-way convergence: evolution, neuroscience, AI suggests fundamental importance
  • Most intelligence may be sophisticated pattern matching with hierarchical associations

Reasoning and Association

  • "Associations all the way down" - most intelligence is advanced pattern matching
  • Sherlock Holmes reasoning explained as long working memory + sophisticated associations
  • Higher-level associations enable complex reasoning chains
  • Models build rich representations by combining information across layers
  • Each layer operates on combinations of all previous tokens

Intelligence Explosion and Research Automation

  • Current bottlenecks: compute for experiments and "taste" for research directions
  • 10x compute could make research 5x faster (0.5 elasticity)
  • Reliability improvements crucial for agent deployment
  • Recursive self-improvement limited by training costs for larger models
  • Research automation more likely through augmenting top researchers than full automation

Career Paths and Hiring in AI

  • Both researchers hired through non-traditional paths (consulting, neuroscience)
  • Agency and "caring an unbelievable amount" more important than credentials
  • Demonstrated ability through public work (blogs, papers, projects) crucial
  • Mentorship from senior researchers enables rapid skill development
  • Cross-disciplinary backgrounds valuable for fresh perspectives
  • "System is not your friend" - proactive self-advocacy necessary

Mechanistic Interpretability Fundamentals

  • Superposition: models pack more features than parameters allow
  • Dictionary learning projects activations to higher-dimensional sparse space
  • Features emerge as directions in activation space
  • Feature splitting: more capacity → more specific features (bird → ravens, eagles, sparrows)
  • Models dramatically underparameterized for internet-scale prediction tasks

Feature Discovery and Analysis

  • Base64 features example: three separate features for different encoding types
  • Features range from low-level (tokens) to high-level (love, sudden scene changes)
  • Automated interpretability needed for superhuman model analysis
  • Feature universality across models suggests fundamental learning patterns
  • Geometry of features enables hierarchical exploration (biology → cells → anthrax)

Safety Applications

  • Dictionary learning for detecting deception circuits in models
  • Unsupervised approach avoids labeling biases of linear probes
  • Ablation studies to remove unwanted behaviors post-training
  • Anomaly detection for novel feature activations
  • Scaling challenges: millions of features in future models require automation

Future of AI Systems

  • Debate over agents vs single large models with adaptive compute
  • Near-term likely modular agents for human understanding and control
  • Long-term may converge to unified systems with dynamic resource allocation
  • Fine-tuning may become obsolete with infinite context
  • End-to-end training on business outcomes theoretically possible but practically difficult

Multimodal and Synthetic Data

  • Language evolved to be learnable - explains LLM success compared to vision
  • Positive transfer between modalities (code training improves reasoning)
  • Synthetic data crucial for future progress - models generating training data
  • Verification functions needed for synthetic data quality
  • Curriculum learning underexplored despite evidence of staged capability development

Scaling Laws and Future Progress

  • Diminishing returns on raw compute scaling
  • Each order of magnitude gives more reliability nines, unlocking agents
  • Still below brain scale in parameters (1T vs 30-300T synapses)
  • Algorithmic improvements could dramatically reduce compute requirements
  • Sample efficiency improves with scale - larger models learn more from same data

Safety and Control Concerns

  • Feature-level control raises value lock-in concerns
  • Need transparency in model constitutions and public input
  • Balance between safety and preserving beneficial capabilities
  • Interpretability provides more precise tools than RLHF
  • Unknown unknowns remain for superhuman capabilities

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome