AI summary by TwinMind
Deep Dive into LLMs like ChatGPT
Andrej Karpathy
3h 31m 23s
Introduction to LLMs
- Speaker aims to provide comprehensive, general audience introduction to large language models (LLMs) like ChatGPT
- Goal: provide mental models for understanding what LLMs are, their capabilities, limitations, and "sharp edges"
- Will cover entire pipeline of how LLMs are built while keeping explanation accessible
- Will discuss cognitive/psychological implications of these tools
How LLMs Are Built: Pre-training Stage
- First stage: Pre-training involves downloading and processing internet data
- Referenced Fine-web dataset from Hugging Face as example of curated training data
- Goal: collect large quantity of high-quality, diverse documents containing knowledge
Data Collection and Processing
- Common Crawl is primary data source (organization scouring internet since 2007)
- As of 2024, Common Crawl has indexed 2.7 billion web pages
- Fine-web dataset is approximately 44 terabytes (could fit on a single hard drive)
- Multiple filtering stages applied to raw data:
- URL filtering: removes malware, spam, marketing, racist, adult sites
- Text extraction: separates content from HTML/CSS markup
- Language filtering: e.g., Fine-web keeps pages with >65% English content
- PII (personally identifiable information) removal
- Deduplication
Tokenization
- LLMs require text to be converted to tokens (finite set of symbols)
- UTF-8 encoding produces raw bits (0s and 1s) but sequence is too long
- Tokenization trades off vocabulary size vs. sequence length
- Byte-pair encoding (BPE) algorithm:
- Starts with individual characters/bytes
- Merges common consecutive pairs into new tokens
- Continues process until desired vocabulary size
- GPT-4 uses 100,277 tokens in its vocabulary
- Demonstrated with tiktoken.org website showing how text breaks into tokens
- Tokens are the "atoms" of text that the model works with
Neural Network Training
- Pre-training objective: predict next token in sequence
- Process:
- Take windows of tokens from dataset (e.g., 4 tokens)
- Feed into neural network to predict next token
- Compare prediction with actual next token
- Update network parameters to improve prediction
- Repeat process across billions of examples
- Neural networks start randomly initialized, gradually learn patterns
- Networks are mathematical expressions mixing inputs with parameters
- Transformer architecture shown as example (85,000 parameters)
- Information flows through network in forward pass to produce predictions
Inference
- After training, models generate text through sampling:
- Start with seed tokens
- Feed into neural network to get probability distribution
- Sample next token based on probabilities
- Add sampled token to sequence
- Repeat process to continue generating
- Stochastic process - different runs produce different outputs
- May occasionally reproduce exact sequences from training data, but typically creates novel combinations
GPT-2 Example
- GPT-2 (2019) was first "recognizably modern" LLM architecture
- Specifications:
- 1.5 billion parameters
- 1,024 token context length
- Trained on ~100 billion tokens
- Cost to train in 2019: ~$40,000; today: ~$100-600 due to improved hardware/software
- Training process shown: loss decreases with each update
- Inference examples gradually improving coherence as training progresses
Compute Requirements
- Training requires specialized hardware (GPUs)
- H100 GPUs cost ~$3/hour per GPU
- Data centers stack multiple GPUs (e.g., 8 GPUs per node)
- Major tech companies building massive GPU clusters
- Example: Elon Musk acquiring 100,000 GPUs for a data center
Base Models
- "Base model" is output of pre-training stage
- Functions as "internet text token simulator"
- Not directly useful as assistant without further training
- Examples of base models:
- GPT-2 (1.5B parameters) from 2019
- Llama 3.1 (405B parameters) from Meta
Base Model Capabilities
- Demonstrated on Llama 3.1 405B base model via hyperbolic.ai
- Acts as text autocomplete, not an assistant
- Memorizes popular content (e.g., Wikipedia articles)
- Makes up information when asked about unknown entities
- Can be prompted to show knowledge through clever prompting
- Knowledge cutoff (e.g., end of 2023 for Llama 3.1)
- In-context learning through few-shot prompting
- Can be turned into simple assistant with prompt engineering
Post-Training Stage: Supervised Fine-Tuning (SFT)
- Takes base model and converts it to assistant through conversations
- Conversations structured as turns between human and assistant
- Tokenization of conversations shown (special tokens for roles)
- Training on conversations teaches model to behave as assistant
SFT Data Creation
- Human labelers create conversations following company guidelines
- InstructGPT paper (2022) described process:
- Labelers hired through Upwork/Scale AI
- Given instructions to be helpful, truthful, harmless
- Create prompts and ideal responses
- Modern datasets (e.g., UltraChat) often use synthetic data with human editing
- Millions of conversations covering diverse topics
Understanding LLM Assistants
- What users interact with is statistical imitation of human labelers
- Not "magical AI" but simulation of humans following labeling instructions
- Example: asking for Paris landmarks gets response similar to what a human labeler would write
LLM Psychology and Limitations
Hallucinations
- LLMs make up information when uncertain
- Root cause: in training data, questions always get confident answers
- Models statistically imitate answer format even when they don't know
- Mitigation strategies:
- Train models to say "I don't know" when uncertain
- Interrogate model on known facts to find knowledge boundaries
- Add examples where correct response is "I don't know"
- Provide tools (e.g., web search)
- Model can emit special tokens to trigger web search
- Search results enter context window as "working memory"
- Model can then reference accurate information
- Train models to say "I don't know" when uncertain
Knowledge Organization
- Knowledge in parameters = vague recollection (like long-term memory)
- Knowledge in context window = working memory (immediately accessible)
- Better results when providing text directly rather than asking model to recall
Self-Knowledge
- Questions about model identity are problematic
- Models have no persistent self, just statistical patterns
- Identity can be programmed through:
- Training data with identity information
- System messages in context window
Computational Capabilities
- Models need tokens to think - computation distributed across tokens
- Fixed amount of computation per token limits complexity per step
- Example: Math problem solving
- Bad: Asking for immediate answer in single token
- Good: Breaking down into steps with intermediate results
- Models struggle with counting, spelling due to tokenization
- Can use tools (code interpreter) to overcome limitations
- Unexplainable failures exist (e.g., comparing 9.11 vs 9.9)
Reinforcement Learning (RL) Stage
- Final major stage in training pipeline
- Analogous to practice problems in textbooks
- Supervised fine-tuning = learning from worked examples
- Reinforcement learning = practicing to find best solutions
RL Process
- For each prompt, generate many candidate solutions
- Evaluate solutions based on correctness
- Train on successful solutions
- Model discovers effective problem-solving strategies on its own
- Deep Seek paper showed importance of RL for reasoning capabilities
Reasoning Models
- RL training produces "thinking models" that show reasoning process
- Models learn to break down problems, try different approaches
- Demonstrate self-correction, verification, multiple solution methods
- Available in models like Deep Seek R1, Claude Sonnet, GPT-4o, Gemini
Reinforcement Learning from Human Feedback (RLHF)
- Approach for unverifiable domains (creative writing, summarization)
- Process:
- Generate multiple outputs for prompt
- Have humans rank outputs
- Train reward model to predict human preferences
- Use reward model for reinforcement learning
- Advantages: improves model quality, easier for humans than writing examples
- Limitations:
- Uses lossy simulation of human judgment
- Models can "game" reward function with adversarial examples
- Can't run indefinitely like true RL
Future Capabilities and Resources
- Multimodality: handling text, audio, images natively
- Agents: long-running tasks with human supervision
- Pervasive integration into tools
- Test-time training and learning
- Resources to follow:
- LMsys.org leaderboard
- AI News newsletter
- X/Twitter
- Where to find models:
- Provider websites (OpenAI, Google)
- Inference providers (Together.ai)
- Hyperbolic.ai for base models
- LM Studio for local running
Conclusion
- LLMs are trained in three stages:
- Pre-training on internet text (knowledge acquisition)
- Supervised fine-tuning on conversations (imitation)
- Reinforcement learning (practice and improvement)
- Models are statistical simulations with limitations
- Use as tools, check work, don't fully trust
- Exciting technology that can dramatically accelerate work
