AI summary by TwinMind
Deep Dive into LLMs like ChatGPT
Andrej Karpathy
3h 31m 23s
Introduction to Large Language Models
- Video aims to provide mental models for understanding LLMs like ChatGPT
- Goals: explain what's behind text box, how words are generated, what users are interacting with
- Will cover entire pipeline of how LLMs are built while keeping accessible to general audience
- Will discuss cognitive/psychological implications of these tools
Pre-training Stage: Building the Foundation
- First stage involves downloading and processing internet text
- Example: Hugging Face's "Fine Web" dataset (44 terabytes)
- Goal: obtain high-quality, diverse documents in large quantities
- Multi-stage filtering process includes:
- URL filtering (removes malware, spam, marketing, racist, adult sites)
- Text extraction (removes HTML markup, keeps only content)
- Language filtering (e.g., keeping pages with >65% English)
- PII (personally identifiable information) removal
- Results in 44TB dataset with 15 trillion tokens
- Raw internet text must be converted to tokens for neural network processing
- Neural networks require one-dimensional sequences of symbols from finite set
- UTF-8 encoding converts text to bits (0s and 1s)
- Bits grouped into bytes (256 possible combinations)
- Byte Pair Encoding (BPE) creates vocabulary of tokens
- GPT-4 uses 100,277 tokens
- Common sequences of bytes/characters merged into single tokens
- Tokenization example shown using "tiktoken" website
- Words/phrases split differently depending on spacing, capitalization
- Models see token IDs, not characters
Neural Network Training
- Neural networks predict next token in sequence based on previous tokens
- Pre-training involves:
- Taking windows of tokens from dataset (e.g., 8,000 tokens)
- Neural network predicts probability distribution for next token
- Compare prediction with actual next token
- Update network parameters to improve prediction
- Neural network internals:
- Transformer architecture mixes inputs with parameters
- Parameters initially random, gradually tuned during training
- Example: GPT-2 (1.5B parameters) shown with 85,000 parameters
- Information flows through attention blocks and multi-layer perceptrons
- Computation requirements:
- Training happens on GPUs (e.g., Nvidia H100)
- Large companies use thousands of GPUs in data centers
- Explains Nvidia's $3.4 trillion valuation - "gold rush" for GPUs
- Example: Elon Musk acquiring 100,000 GPUs for AI development
Inference: Generating Text
- After training, model generates text through sampling:
- Start with seed tokens (prompt)
- Model outputs probability distribution for next token
- Sample from distribution (like flipping biased coin)
- Add sampled token to sequence and repeat
- Stochastic process creates different outputs each time
- May occasionally reproduce training data but usually creates novel text
GPT-2 Example and Modern LLM Evolution
- GPT-2 (2019): 1.5B parameters, 1,024 token context, trained on 100B tokens
- Cost $40,000 to train in 2019, now costs ~$100-600
- Modern models have hundreds of billions of parameters, context lengths of hundreds of thousands, trained on trillions of tokens
- Training visualization shown: loss decreases as model improves
- Base models like Llama 3 (Meta) have been released with open weights
- Llama 3.1 405B parameter model is state-of-the-art open model
- Base models are "internet document simulators" (not yet assistants)
- Base model behaviors:
- Can memorize and regurgitate training data (e.g., Wikipedia articles)
- Makes educated guesses about unknown facts (hallucinations)
- Has in-context learning abilities (can learn patterns from examples)
- Can be prompted to act like an assistant with clever prompting
Post-Training Stage: Creating Assistants
- Base models need fine-tuning to become helpful assistants
- Supervised Fine-Tuning (SFT):
- Train on conversations between humans and assistants
- Conversations encoded into token sequences with special markers
- Human labelers create ideal assistant responses following guidelines
- Model learns to imitate helpful, truthful, harmless responses
- SFT datasets:
- InstructGPT (OpenAI 2022) pioneered this approach
- Modern datasets like UltraChat contain millions of conversations
- Increasingly use existing LLMs to help generate data with human editing
- What users interact with in ChatGPT:
- Statistical simulation of human labelers following company guidelines
- Not magical AI but imitation of human experts' responses
LLM Psychology and Limitations
- Hallucinations: models make up facts when uncertain
- Root cause: training data teaches confident answering style
- Models don't naturally express uncertainty
- Mitigation: include examples of "I don't know" in training data
- Companies probe model knowledge boundaries to teach appropriate refusal
- Tool use as mitigation for limitations:
- Web search: model emits special tokens to trigger search
- Results added to context window (working memory)
- Allows model to access current information beyond training cutoff
- Code interpreter: runs code to perform calculations reliably
- Web search: model emits special tokens to trigger search
- Knowledge representation:
- Parameters = vague recollection (long-term memory)
- Context window = working memory (immediately accessible)
- Best results come from putting information in context window
- Self-knowledge limitations:
- Models have no persistent existence or identity
- Companies hardcode identity information via:
- Training data with identity questions/answers
- System messages with model specifications
Computational Capabilities and Limitations
- Models need tokens to think - computation distributed across tokens
- Each token prediction has limited computation (fixed neural network depth)
- Complex reasoning must be broken into steps across multiple tokens
- Example: math problem solutions should show work, not jump to answer
- Limitations in specific tasks:
- Counting: models struggle with counting items in context
- Spelling/character manipulation: models see tokens, not characters
- Simple comparisons sometimes fail (e.g., 9.11 vs 9.9)
- Tools help overcome limitations:
- Code interpreter for calculations and string manipulation
- Web search for factual information
Reinforcement Learning: Teaching Models to Think
- Third major stage in training pipeline
- Analogous to practice problems in education
- Pre-training = reading background material
- SFT = studying worked examples
- RL = solving practice problems independently
- Process:
- Generate many different solutions to problems
- Keep solutions that reach correct answers
- Train model to produce more solutions like successful ones
- Benefits:
- Model discovers its own reasoning strategies
- Develops "chain of thought" approaches
- Solutions become longer but more accurate
- Can discover novel problem-solving approaches
- Deep Seek R1 example:
- Shows thinking process: trying different approaches, checking work
- Available as open-weights model through various providers
- Similar to Claude "Sonnet" and GPT-4o "thinking" models
- Comparison to AlphaGo:
- RL in Go led to superhuman play (e.g., famous "Move 37")
- RL in LLMs could potentially discover novel reasoning strategies
Reinforcement Learning from Human Feedback (RLHF)
- Used for unverifiable domains (creative writing, summarization)
- Process:
- Generate multiple outputs for each prompt
- Humans rank outputs from best to worst
- Train reward model to predict human preferences
- Use RL against reward model instead of directly against humans
- Advantages:
- Works in domains without objective metrics
- Easier for humans to rank than generate ideal responses
- Empirically improves model quality
- Limitations:
- Uses lossy simulation of human judgment
- Reward models can be "gamed" by RL finding adversarial inputs
- Must be stopped after limited iterations before degradation
Future Developments and Resources
- Future capabilities:
- Multimodal models (text, audio, images, video)
- Long-running agents for complex tasks
- More pervasive and invisible integration
- Computer control (keyboard/mouse actions)
- Test-time training for continuous learning
- Resources to follow:
- LMSys Leaderboard (ElMarine) for model rankings
- AI News newsletter
- X/Twitter for latest developments
- Where to find models:
- Provider websites (OpenAI, Google)
- Inference providers (Together.ai)
- Hyperbolic for base models
- LM Studio for local running of smaller models
Conclusion
- What happens when using ChatGPT:
- Query tokenized and processed through neural network
- Response is simulation of human labeler following company guidelines
- Thinking models add emergent reasoning capabilities
- LLMs are powerful tools but have limitations:
- Swiss cheese capabilities (excellent in some areas, poor in others)
- Can hallucinate or make simple mistakes
- Should be used with verification
- Exciting time for AI development, with models becoming increasingly capable
