AI summary by TwinMind
Sholto Douglas & Trenton Bricken - How LLMs Actually Think
Dwarkesh Patel
3h 13m 13s
Context Windows and Their Importance
- Long context capabilities dramatically underappreciated despite allowing million+ token inputs
- Solves "onboarding problem" for models - can instantly understand entire codebases
- Shows step-up in intelligence comparable to huge model scale increases
- Models learn languages in-context better than human experts over months
- Example: esoteric human language not in training data, model learns from context alone
- Creates superhuman capability in information integration (humans can't hold million tokens in context)
In-Context Learning Mechanisms
- In-context learning similar to gradient descent in attention operations
- Model performs meta-learning - learning how to learn from context examples
- Shows exact correspondence between gradient descent steps and attention layers
- Enables dramatic sample efficiency improvements with scale
- Training on long context tasks forces models to develop meta-learning abilities
- Creates flexible, adaptive intelligence through better context utilization
Long Horizon Tasks vs Reliability
- AI agents haven't succeeded due to reliability issues, not long context limitations
- Need "nines of reliability" - small probability failures compound over long task chains
- 99% → 90% success rate makes multi-step tasks much less likely to succeed
- Academic evals focus on single problems, not chained multi-hour/multi-day tasks
- Sweet Bench represents progress toward realistic long-horizon evaluation
- Economic impact requires understanding success rates over different time horizons
Attention Mechanisms and Computational Costs
- Quadratic attention cost often dominated by MLP blocks in practice
- At inference, attention is linear (one query against many KV vectors)
- Actual bottlenecks differ from theoretical scaling concerns
- Google achieving million-token windows suggests quadratic costs manageable
- Research community overemphasized attention scaling problems
Forward Pass vs Backward Pass Learning
- Trend toward more learning happening in forward pass (in-context) vs backward pass (training)
- Represents progression from bottom-up evolutionary learning to top-down reasoning
- Chain of Thought as adaptive compute - more forward passes for harder problems
- Models can transmit information through KV cache beyond just output tokens
- Steganography possible in attention mechanisms during reasoning
Brain-AI Analogies and Attention
- Cerebellum circuit matches attention operation almost exactly
- Associative memory algorithm from 1980s identical to cerebellar architecture
- Same architecture appears across evolution (fruit flies to humans)
- Three-way convergence: evolution, neuroscience, AI suggests fundamental importance
- Most intelligence may be sophisticated pattern matching with hierarchical associations
Reasoning and Association
- "Associations all the way down" - most intelligence is advanced pattern matching
- Sherlock Holmes reasoning explained as long working memory + sophisticated associations
- Higher-level associations enable complex reasoning chains
- Models build rich representations by combining information across layers
- Each layer operates on combinations of all previous tokens
Intelligence Explosion and Research Automation
- Current bottlenecks: compute for experiments and "taste" for research directions
- 10x compute could make research 5x faster (0.5 elasticity)
- Reliability improvements crucial for agent deployment
- Recursive self-improvement limited by training costs for larger models
- Research automation more likely through augmenting top researchers than full automation
Career Paths and Hiring in AI
- Both researchers hired through non-traditional paths (consulting, neuroscience)
- Agency and "caring an unbelievable amount" more important than credentials
- Demonstrated ability through public work (blogs, papers, projects) crucial
- Mentorship from senior researchers enables rapid skill development
- Cross-disciplinary backgrounds valuable for fresh perspectives
- "System is not your friend" - proactive self-advocacy necessary
Mechanistic Interpretability Fundamentals
- Superposition: models pack more features than parameters allow
- Dictionary learning projects activations to higher-dimensional sparse space
- Features emerge as directions in activation space
- Feature splitting: more capacity → more specific features (bird → ravens, eagles, sparrows)
- Models dramatically underparameterized for internet-scale prediction tasks
Feature Discovery and Analysis
- Base64 features example: three separate features for different encoding types
- Features range from low-level (tokens) to high-level (love, sudden scene changes)
- Automated interpretability needed for superhuman model analysis
- Feature universality across models suggests fundamental learning patterns
- Geometry of features enables hierarchical exploration (biology → cells → anthrax)
Safety Applications
- Dictionary learning for detecting deception circuits in models
- Unsupervised approach avoids labeling biases of linear probes
- Ablation studies to remove unwanted behaviors post-training
- Anomaly detection for novel feature activations
- Scaling challenges: millions of features in future models require automation
Future of AI Systems
- Debate over agents vs single large models with adaptive compute
- Near-term likely modular agents for human understanding and control
- Long-term may converge to unified systems with dynamic resource allocation
- Fine-tuning may become obsolete with infinite context
- End-to-end training on business outcomes theoretically possible but practically difficult
Multimodal and Synthetic Data
- Language evolved to be learnable - explains LLM success compared to vision
- Positive transfer between modalities (code training improves reasoning)
- Synthetic data crucial for future progress - models generating training data
- Verification functions needed for synthetic data quality
- Curriculum learning underexplored despite evidence of staged capability development
Scaling Laws and Future Progress
- Diminishing returns on raw compute scaling
- Each order of magnitude gives more reliability nines, unlocking agents
- Still below brain scale in parameters (1T vs 30-300T synapses)
- Algorithmic improvements could dramatically reduce compute requirements
- Sample efficiency improves with scale - larger models learn more from same data
Safety and Control Concerns
- Feature-level control raises value lock-in concerns
- Need transparency in model constitutions and public input
- Balance between safety and preserving beneficial capabilities
- Interpretability provides more precise tools than RLHF
- Unknown unknowns remain for superhuman capabilities
