AI summary by TwinMind

Deep Dive into LLMs like ChatGPT

Andrej Karpathy

3h 31m 23s

Introduction to Large Language Models

  • Video aims to provide mental models for understanding LLMs like ChatGPT
  • Goals: explain what's behind text box, how words are generated, what users are interacting with
  • Will cover entire pipeline of how LLMs are built while keeping accessible to general audience
  • Will discuss cognitive/psychological implications of these tools

Pre-training Stage: Building the Foundation

  • First stage involves downloading and processing internet text
  • Example: Hugging Face's "Fine Web" dataset (44 terabytes)
    • Goal: obtain high-quality, diverse documents in large quantities
    • Multi-stage filtering process includes:
      • URL filtering (removes malware, spam, marketing, racist, adult sites)
      • Text extraction (removes HTML markup, keeps only content)
      • Language filtering (e.g., keeping pages with >65% English)
      • PII (personally identifiable information) removal
    • Results in 44TB dataset with 15 trillion tokens
  • Raw internet text must be converted to tokens for neural network processing
    • Neural networks require one-dimensional sequences of symbols from finite set
    • UTF-8 encoding converts text to bits (0s and 1s)
    • Bits grouped into bytes (256 possible combinations)
    • Byte Pair Encoding (BPE) creates vocabulary of tokens
      • GPT-4 uses 100,277 tokens
      • Common sequences of bytes/characters merged into single tokens
  • Tokenization example shown using "tiktoken" website
    • Words/phrases split differently depending on spacing, capitalization
    • Models see token IDs, not characters

Neural Network Training

  • Neural networks predict next token in sequence based on previous tokens
  • Pre-training involves:
    • Taking windows of tokens from dataset (e.g., 8,000 tokens)
    • Neural network predicts probability distribution for next token
    • Compare prediction with actual next token
    • Update network parameters to improve prediction
  • Neural network internals:
    • Transformer architecture mixes inputs with parameters
    • Parameters initially random, gradually tuned during training
    • Example: GPT-2 (1.5B parameters) shown with 85,000 parameters
    • Information flows through attention blocks and multi-layer perceptrons
  • Computation requirements:
    • Training happens on GPUs (e.g., Nvidia H100)
    • Large companies use thousands of GPUs in data centers
    • Explains Nvidia's $3.4 trillion valuation - "gold rush" for GPUs
    • Example: Elon Musk acquiring 100,000 GPUs for AI development

Inference: Generating Text

  • After training, model generates text through sampling:
    • Start with seed tokens (prompt)
    • Model outputs probability distribution for next token
    • Sample from distribution (like flipping biased coin)
    • Add sampled token to sequence and repeat
  • Stochastic process creates different outputs each time
  • May occasionally reproduce training data but usually creates novel text

GPT-2 Example and Modern LLM Evolution

  • GPT-2 (2019): 1.5B parameters, 1,024 token context, trained on 100B tokens
    • Cost $40,000 to train in 2019, now costs ~$100-600
    • Modern models have hundreds of billions of parameters, context lengths of hundreds of thousands, trained on trillions of tokens
  • Training visualization shown: loss decreases as model improves
  • Base models like Llama 3 (Meta) have been released with open weights
    • Llama 3.1 405B parameter model is state-of-the-art open model
    • Base models are "internet document simulators" (not yet assistants)
  • Base model behaviors:
    • Can memorize and regurgitate training data (e.g., Wikipedia articles)
    • Makes educated guesses about unknown facts (hallucinations)
    • Has in-context learning abilities (can learn patterns from examples)
    • Can be prompted to act like an assistant with clever prompting

Post-Training Stage: Creating Assistants

  • Base models need fine-tuning to become helpful assistants
  • Supervised Fine-Tuning (SFT):
    • Train on conversations between humans and assistants
    • Conversations encoded into token sequences with special markers
    • Human labelers create ideal assistant responses following guidelines
    • Model learns to imitate helpful, truthful, harmless responses
  • SFT datasets:
    • InstructGPT (OpenAI 2022) pioneered this approach
    • Modern datasets like UltraChat contain millions of conversations
    • Increasingly use existing LLMs to help generate data with human editing
  • What users interact with in ChatGPT:
    • Statistical simulation of human labelers following company guidelines
    • Not magical AI but imitation of human experts' responses

LLM Psychology and Limitations

  • Hallucinations: models make up facts when uncertain
    • Root cause: training data teaches confident answering style
    • Models don't naturally express uncertainty
    • Mitigation: include examples of "I don't know" in training data
    • Companies probe model knowledge boundaries to teach appropriate refusal
  • Tool use as mitigation for limitations:
    • Web search: model emits special tokens to trigger search
      • Results added to context window (working memory)
      • Allows model to access current information beyond training cutoff
    • Code interpreter: runs code to perform calculations reliably
  • Knowledge representation:
    • Parameters = vague recollection (long-term memory)
    • Context window = working memory (immediately accessible)
    • Best results come from putting information in context window
  • Self-knowledge limitations:
    • Models have no persistent existence or identity
    • Companies hardcode identity information via:
      • Training data with identity questions/answers
      • System messages with model specifications

Computational Capabilities and Limitations

  • Models need tokens to think - computation distributed across tokens
    • Each token prediction has limited computation (fixed neural network depth)
    • Complex reasoning must be broken into steps across multiple tokens
    • Example: math problem solutions should show work, not jump to answer
  • Limitations in specific tasks:
    • Counting: models struggle with counting items in context
    • Spelling/character manipulation: models see tokens, not characters
    • Simple comparisons sometimes fail (e.g., 9.11 vs 9.9)
  • Tools help overcome limitations:
    • Code interpreter for calculations and string manipulation
    • Web search for factual information

Reinforcement Learning: Teaching Models to Think

  • Third major stage in training pipeline
  • Analogous to practice problems in education
    • Pre-training = reading background material
    • SFT = studying worked examples
    • RL = solving practice problems independently
  • Process:
    • Generate many different solutions to problems
    • Keep solutions that reach correct answers
    • Train model to produce more solutions like successful ones
  • Benefits:
    • Model discovers its own reasoning strategies
    • Develops "chain of thought" approaches
    • Solutions become longer but more accurate
    • Can discover novel problem-solving approaches
  • Deep Seek R1 example:
    • Shows thinking process: trying different approaches, checking work
    • Available as open-weights model through various providers
    • Similar to Claude "Sonnet" and GPT-4o "thinking" models
  • Comparison to AlphaGo:
    • RL in Go led to superhuman play (e.g., famous "Move 37")
    • RL in LLMs could potentially discover novel reasoning strategies

Reinforcement Learning from Human Feedback (RLHF)

  • Used for unverifiable domains (creative writing, summarization)
  • Process:
    • Generate multiple outputs for each prompt
    • Humans rank outputs from best to worst
    • Train reward model to predict human preferences
    • Use RL against reward model instead of directly against humans
  • Advantages:
    • Works in domains without objective metrics
    • Easier for humans to rank than generate ideal responses
    • Empirically improves model quality
  • Limitations:
    • Uses lossy simulation of human judgment
    • Reward models can be "gamed" by RL finding adversarial inputs
    • Must be stopped after limited iterations before degradation

Future Developments and Resources

  • Future capabilities:
    • Multimodal models (text, audio, images, video)
    • Long-running agents for complex tasks
    • More pervasive and invisible integration
    • Computer control (keyboard/mouse actions)
    • Test-time training for continuous learning
  • Resources to follow:
    • LMSys Leaderboard (ElMarine) for model rankings
    • AI News newsletter
    • X/Twitter for latest developments
  • Where to find models:
    • Provider websites (OpenAI, Google)
    • Inference providers (Together.ai)
    • Hyperbolic for base models
    • LM Studio for local running of smaller models

Conclusion

  • What happens when using ChatGPT:
    • Query tokenized and processed through neural network
    • Response is simulation of human labeler following company guidelines
    • Thinking models add emergent reasoning capabilities
  • LLMs are powerful tools but have limitations:
    • Swiss cheese capabilities (excellent in some areas, poor in others)
    • Can hallucinate or make simple mistakes
    • Should be used with verification
  • Exciting time for AI development, with models becoming increasingly capable

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome