AI summary by TwinMind

Deep Dive into LLMs like ChatGPT

Andrej Karpathy

3h 31m 23s

Introduction to LLMs

  • Speaker aims to provide comprehensive, general audience introduction to large language models (LLMs) like ChatGPT
  • Goal: provide mental models for understanding what LLMs are, their capabilities, limitations, and "sharp edges"
  • Will cover entire pipeline of how LLMs are built while keeping explanation accessible
  • Will discuss cognitive/psychological implications of these tools

How LLMs Are Built: Pre-training Stage

  • First stage: Pre-training involves downloading and processing internet data
  • Referenced Fine-web dataset from Hugging Face as example of curated training data
  • Goal: collect large quantity of high-quality, diverse documents containing knowledge

Data Collection and Processing

  • Common Crawl is primary data source (organization scouring internet since 2007)
  • As of 2024, Common Crawl has indexed 2.7 billion web pages
  • Fine-web dataset is approximately 44 terabytes (could fit on a single hard drive)
  • Multiple filtering stages applied to raw data:
    • URL filtering: removes malware, spam, marketing, racist, adult sites
    • Text extraction: separates content from HTML/CSS markup
    • Language filtering: e.g., Fine-web keeps pages with >65% English content
    • PII (personally identifiable information) removal
    • Deduplication

Tokenization

  • LLMs require text to be converted to tokens (finite set of symbols)
  • UTF-8 encoding produces raw bits (0s and 1s) but sequence is too long
  • Tokenization trades off vocabulary size vs. sequence length
  • Byte-pair encoding (BPE) algorithm:
    • Starts with individual characters/bytes
    • Merges common consecutive pairs into new tokens
    • Continues process until desired vocabulary size
  • GPT-4 uses 100,277 tokens in its vocabulary
  • Demonstrated with tiktoken.org website showing how text breaks into tokens
  • Tokens are the "atoms" of text that the model works with

Neural Network Training

  • Pre-training objective: predict next token in sequence
  • Process:
    • Take windows of tokens from dataset (e.g., 4 tokens)
    • Feed into neural network to predict next token
    • Compare prediction with actual next token
    • Update network parameters to improve prediction
    • Repeat process across billions of examples
  • Neural networks start randomly initialized, gradually learn patterns
  • Networks are mathematical expressions mixing inputs with parameters
  • Transformer architecture shown as example (85,000 parameters)
  • Information flows through network in forward pass to produce predictions

Inference

  • After training, models generate text through sampling:
    • Start with seed tokens
    • Feed into neural network to get probability distribution
    • Sample next token based on probabilities
    • Add sampled token to sequence
    • Repeat process to continue generating
  • Stochastic process - different runs produce different outputs
  • May occasionally reproduce exact sequences from training data, but typically creates novel combinations

GPT-2 Example

  • GPT-2 (2019) was first "recognizably modern" LLM architecture
  • Specifications:
    • 1.5 billion parameters
    • 1,024 token context length
    • Trained on ~100 billion tokens
  • Cost to train in 2019: ~$40,000; today: ~$100-600 due to improved hardware/software
  • Training process shown: loss decreases with each update
  • Inference examples gradually improving coherence as training progresses

Compute Requirements

  • Training requires specialized hardware (GPUs)
  • H100 GPUs cost ~$3/hour per GPU
  • Data centers stack multiple GPUs (e.g., 8 GPUs per node)
  • Major tech companies building massive GPU clusters
    • Example: Elon Musk acquiring 100,000 GPUs for a data center

Base Models

  • "Base model" is output of pre-training stage
  • Functions as "internet text token simulator"
  • Not directly useful as assistant without further training
  • Examples of base models:
    • GPT-2 (1.5B parameters) from 2019
    • Llama 3.1 (405B parameters) from Meta

Base Model Capabilities

  • Demonstrated on Llama 3.1 405B base model via hyperbolic.ai
  • Acts as text autocomplete, not an assistant
  • Memorizes popular content (e.g., Wikipedia articles)
  • Makes up information when asked about unknown entities
  • Can be prompted to show knowledge through clever prompting
  • Knowledge cutoff (e.g., end of 2023 for Llama 3.1)
  • In-context learning through few-shot prompting
  • Can be turned into simple assistant with prompt engineering

Post-Training Stage: Supervised Fine-Tuning (SFT)

  • Takes base model and converts it to assistant through conversations
  • Conversations structured as turns between human and assistant
  • Tokenization of conversations shown (special tokens for roles)
  • Training on conversations teaches model to behave as assistant

SFT Data Creation

  • Human labelers create conversations following company guidelines
  • InstructGPT paper (2022) described process:
    • Labelers hired through Upwork/Scale AI
    • Given instructions to be helpful, truthful, harmless
    • Create prompts and ideal responses
  • Modern datasets (e.g., UltraChat) often use synthetic data with human editing
  • Millions of conversations covering diverse topics

Understanding LLM Assistants

  • What users interact with is statistical imitation of human labelers
  • Not "magical AI" but simulation of humans following labeling instructions
  • Example: asking for Paris landmarks gets response similar to what a human labeler would write

LLM Psychology and Limitations

Hallucinations

  • LLMs make up information when uncertain
  • Root cause: in training data, questions always get confident answers
  • Models statistically imitate answer format even when they don't know
  • Mitigation strategies:
    1. Train models to say "I don't know" when uncertain
      • Interrogate model on known facts to find knowledge boundaries
      • Add examples where correct response is "I don't know"
    2. Provide tools (e.g., web search)
      • Model can emit special tokens to trigger web search
      • Search results enter context window as "working memory"
      • Model can then reference accurate information

Knowledge Organization

  • Knowledge in parameters = vague recollection (like long-term memory)
  • Knowledge in context window = working memory (immediately accessible)
  • Better results when providing text directly rather than asking model to recall

Self-Knowledge

  • Questions about model identity are problematic
  • Models have no persistent self, just statistical patterns
  • Identity can be programmed through:
    • Training data with identity information
    • System messages in context window

Computational Capabilities

  • Models need tokens to think - computation distributed across tokens
  • Fixed amount of computation per token limits complexity per step
  • Example: Math problem solving
    • Bad: Asking for immediate answer in single token
    • Good: Breaking down into steps with intermediate results
  • Models struggle with counting, spelling due to tokenization
  • Can use tools (code interpreter) to overcome limitations
  • Unexplainable failures exist (e.g., comparing 9.11 vs 9.9)

Reinforcement Learning (RL) Stage

  • Final major stage in training pipeline
  • Analogous to practice problems in textbooks
  • Supervised fine-tuning = learning from worked examples
  • Reinforcement learning = practicing to find best solutions

RL Process

  • For each prompt, generate many candidate solutions
  • Evaluate solutions based on correctness
  • Train on successful solutions
  • Model discovers effective problem-solving strategies on its own
  • Deep Seek paper showed importance of RL for reasoning capabilities

Reasoning Models

  • RL training produces "thinking models" that show reasoning process
  • Models learn to break down problems, try different approaches
  • Demonstrate self-correction, verification, multiple solution methods
  • Available in models like Deep Seek R1, Claude Sonnet, GPT-4o, Gemini

Reinforcement Learning from Human Feedback (RLHF)

  • Approach for unverifiable domains (creative writing, summarization)
  • Process:
    1. Generate multiple outputs for prompt
    2. Have humans rank outputs
    3. Train reward model to predict human preferences
    4. Use reward model for reinforcement learning
  • Advantages: improves model quality, easier for humans than writing examples
  • Limitations:
    • Uses lossy simulation of human judgment
    • Models can "game" reward function with adversarial examples
    • Can't run indefinitely like true RL

Future Capabilities and Resources

  • Multimodality: handling text, audio, images natively
  • Agents: long-running tasks with human supervision
  • Pervasive integration into tools
  • Test-time training and learning
  • Resources to follow:
    • LMsys.org leaderboard
    • AI News newsletter
    • X/Twitter
  • Where to find models:
    • Provider websites (OpenAI, Google)
    • Inference providers (Together.ai)
    • Hyperbolic.ai for base models
    • LM Studio for local running

Conclusion

  • LLMs are trained in three stages:
    1. Pre-training on internet text (knowledge acquisition)
    2. Supervised fine-tuning on conversations (imitation)
    3. Reinforcement learning (practice and improvement)
  • Models are statistical simulations with limitations
  • Use as tools, check work, don't fully trust
  • Exciting technology that can dramatically accelerate work

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome