AI summary by TwinMind

How I use LLMs

Andrej Karpathy

2h 11m 12s

Introduction to LLMs and ChatGPT

  • Speaker continuing general audience series on large language models
    • Previous video covered underhood fundamentals of training and cognition
    • This video focuses on practical applications and examples
  • Current LLM ecosystem overview (as of 2025)
    • ChatGPT by OpenAI as the "Original Gangster" incumbent (most feature-rich)
    • Other major players:
      • Big Tech alternatives: Gemini (Google), Claude (Anthropic), Copilot (Microsoft)
      • Startups: Grok (xAI/Elon Musk's company), DeepSeek (Chinese), LeChatMistral (French)
    • Leaderboards to track model performance:
      • Chatbot Arena (ELO scores)
      • LMSYS leaderboard from Scale

Understanding Token Mechanics

  • Basic interaction: user provides text, model returns text
    • Example: Asking for a haiku about being a language model
  • Under the hood: text is converted to tokens
    • Demonstrated with TikTokenizer tool
    • Example query "Write a haiku about what it's like to be a large language model" = 15 tokens
    • Model response = 19 tokens
  • Conversation format includes special tokens for user/assistant roles
    • Tokens form a one-dimensional sequence in the context window
    • Both user and model contribute to building out the token stream

Managing Conversations and Models

  • Best practices for conversation management:
    • Start a new chat when switching topics to reset context window
    • Tokens in context window are expensive resources
      • Can distract the model from current task
      • Slow down token generation
  • Model selection is important
    • Different pricing tiers offer access to different models
    • ChatGPT pricing structure:
      • Free tier: GPT-4o mini
      • Plus ($20/month): 80 messages/3 hours with GPT-4o
      • Pro ($100/month): Unlimited GPT-4o and additional features
    • Similar tiered access across providers (Claude, Gemini, Grok)

Thinking Models

  • Third stage of training (after pre-training and supervised fine-tuning)
    • Models discover thinking strategies through reinforcement learning
    • Resembles inner monologue of problem-solving
  • OpenAI's thinking models (all start with "o")
    • o1, o3 mini, o3 mini high, o1 Pro, and o1 Promo
    • Requires longer generation time but yields higher accuracy
  • Example: Programming problem with gradient check
    • Regular GPT-4o couldn't solve the issue
    • o1 Pro (thinking model) successfully identified parameter mismatch after 1 minute
    • Other models (Claude, Gemini, Grok) also solved the problem
  • DeepSeek R1 shows raw thinking process
    • Available through Perplexity
    • Shows step-by-step reasoning

Tool Use: Internet Search

  • Internet search allows models to access recent information
    • Example: Asking about White Lotus Season 3 release schedule
    • Model creates search query, visits webpages, extracts information
  • Tool use mechanics:
    • Model emits special "search internet" token
    • Application performs search, adds results to context window
    • Model references this information to answer question
  • Different platforms have varying search capabilities:
    • Perplexity: Early adopter of search integration
    • ChatGPT: Now includes search button
    • Claude: Currently lacks internet search
    • Gemini: Some models have search (2.0 Flash) but not all (2.0 Pro)
  • Practical search use cases demonstrated:
    • Stock market holiday schedules
    • Film/TV production locations
    • Service offerings from companies
    • Product launches and rumors
    • Current events and trending topics

Deep Research

  • Combination of internet search and thinking for comprehensive research
    • Available in ChatGPT Pro tier ($200/month)
    • Takes 5-10 minutes to complete
    • Examines multiple sources, creates detailed report with citations
  • Example: Researching Ca-AKG (health supplement)
    • Model spent 5 minutes reviewing 27 sources
    • Produced detailed report with scientific research, mechanisms of action, safety concerns
  • Similar offerings from other platforms:
    • Perplexity's Deep Research
    • Grok's Deep Search
  • Practical applications demonstrated:
    • Comparing browsers (Brave vs. Arc) for privacy
    • Researching life extension in mice
    • Creating comparative tables of LLM labs

Document Analysis

  • Models can analyze uploaded documents
    • PDFs, research papers, books, etc.
    • Text extracted and loaded into context window
  • Example: Reading papers with LLMs
    • Asking for summaries and explanations of complex papers
    • Helps understand papers outside one's expertise
  • Reading books with LLMs
    • Example: The Wealth of Nations by Adam Smith
    • Copy-pasting chapters and asking for summaries, explanations
    • Increases retention and understanding
    • Particularly useful for older texts or specialized fields

Python Interpreter and Programming

  • Models can write and execute code
    • Used for calculations, data analysis, visualizations
    • Model writes code, application executes it, returns results
  • Example: Complex multiplication
    • Simple calculations done directly by model
    • Complex calculations trigger Python interpreter
    • Different models have different tool access (some hallucinate answers)
  • Advanced Data Analysis in ChatGPT
    • Creates visualizations and performs data analysis
    • Example: Analyzing OpenAI's valuation over time
    • Warning: Models can make implicit assumptions or errors in code
  • Cursor app for professional coding
    • Integrates with Claude 3.7 Sonnet
    • Has context of all files in directory
    • Example: Building Tic-Tac-Toe game with confetti effect
    • "Vibe coding" - giving high-level commands to the model

Multimodal Capabilities

  • Speech/Audio capabilities:
    • Input: Converting speech to text
      • Mobile apps have built-in functionality
      • Desktop requires third-party tools (Super Whisper, Whisper Flow)
    • Output: Text-to-speech for reading responses
    • True audio processing (Advanced Voice Mode)
      • Handles audio natively within model
      • Allows for voice character changes, storytelling
  • Image capabilities:
    • Input: Analyzing uploaded images
      • Example: Nutrition label analysis
      • Blood test results interpretation
      • Mathematical expressions
      • Toothpaste ingredient analysis
      • Meme explanation
    • Output: Image generation with DALL-E
      • Can create images based on prompts
      • Used for content creation, thumbnails
  • Video capabilities:
    • Mobile app can analyze video through camera
    • Can identify objects, read text, describe scenes
    • Video generation tools rapidly evolving (Sora, Gen2, etc.)
  • Notebook LM from Google
    • Creates custom podcasts from uploaded documents
    • Interactive mode allows questions during playback

Quality of Life Features

  • Memory feature in ChatGPT
    • Retains information about user preferences across conversations
    • Creates database of knowledge about the user
    • Example: Remembering opinion about peak Hollywood era
  • Custom instructions
    • Modify how model speaks to you
    • Set personal preferences, identity, language learning settings
  • Custom GPTs
    • Save specific prompts for repeated tasks
    • Example: Korean vocabulary extractor
      • Converts sentences to flashcard format
    • Example: Detailed translator
      • Breaks down sentences part by part
    • Example: OCR + translation for TV subtitles
      • Processes screenshots, extracts text, translates

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome