AI summary by TwinMind
How I use LLMs
Andrej Karpathy
2h 11m 12s
Introduction to LLMs and ChatGPT
- Speaker continuing general audience series on large language models
- Previous video covered underhood fundamentals of training and cognition
- This video focuses on practical applications and examples
- Current LLM ecosystem overview (as of 2025)
- ChatGPT by OpenAI as the "Original Gangster" incumbent (most feature-rich)
- Other major players:
- Big Tech alternatives: Gemini (Google), Claude (Anthropic), Copilot (Microsoft)
- Startups: Grok (xAI/Elon Musk's company), DeepSeek (Chinese), LeChatMistral (French)
- Leaderboards to track model performance:
- Chatbot Arena (ELO scores)
- LMSYS leaderboard from Scale
Understanding Token Mechanics
- Basic interaction: user provides text, model returns text
- Example: Asking for a haiku about being a language model
- Under the hood: text is converted to tokens
- Demonstrated with TikTokenizer tool
- Example query "Write a haiku about what it's like to be a large language model" = 15 tokens
- Model response = 19 tokens
- Conversation format includes special tokens for user/assistant roles
- Tokens form a one-dimensional sequence in the context window
- Both user and model contribute to building out the token stream
Managing Conversations and Models
- Best practices for conversation management:
- Start a new chat when switching topics to reset context window
- Tokens in context window are expensive resources
- Can distract the model from current task
- Slow down token generation
- Model selection is important
- Different pricing tiers offer access to different models
- ChatGPT pricing structure:
- Free tier: GPT-4o mini
- Plus ($20/month): 80 messages/3 hours with GPT-4o
- Pro ($100/month): Unlimited GPT-4o and additional features
- Similar tiered access across providers (Claude, Gemini, Grok)
Thinking Models
- Third stage of training (after pre-training and supervised fine-tuning)
- Models discover thinking strategies through reinforcement learning
- Resembles inner monologue of problem-solving
- OpenAI's thinking models (all start with "o")
- o1, o3 mini, o3 mini high, o1 Pro, and o1 Promo
- Requires longer generation time but yields higher accuracy
- Example: Programming problem with gradient check
- Regular GPT-4o couldn't solve the issue
- o1 Pro (thinking model) successfully identified parameter mismatch after 1 minute
- Other models (Claude, Gemini, Grok) also solved the problem
- DeepSeek R1 shows raw thinking process
- Available through Perplexity
- Shows step-by-step reasoning
Tool Use: Internet Search
- Internet search allows models to access recent information
- Example: Asking about White Lotus Season 3 release schedule
- Model creates search query, visits webpages, extracts information
- Tool use mechanics:
- Model emits special "search internet" token
- Application performs search, adds results to context window
- Model references this information to answer question
- Different platforms have varying search capabilities:
- Perplexity: Early adopter of search integration
- ChatGPT: Now includes search button
- Claude: Currently lacks internet search
- Gemini: Some models have search (2.0 Flash) but not all (2.0 Pro)
- Practical search use cases demonstrated:
- Stock market holiday schedules
- Film/TV production locations
- Service offerings from companies
- Product launches and rumors
- Current events and trending topics
Deep Research
- Combination of internet search and thinking for comprehensive research
- Available in ChatGPT Pro tier ($200/month)
- Takes 5-10 minutes to complete
- Examines multiple sources, creates detailed report with citations
- Example: Researching Ca-AKG (health supplement)
- Model spent 5 minutes reviewing 27 sources
- Produced detailed report with scientific research, mechanisms of action, safety concerns
- Similar offerings from other platforms:
- Perplexity's Deep Research
- Grok's Deep Search
- Practical applications demonstrated:
- Comparing browsers (Brave vs. Arc) for privacy
- Researching life extension in mice
- Creating comparative tables of LLM labs
Document Analysis
- Models can analyze uploaded documents
- PDFs, research papers, books, etc.
- Text extracted and loaded into context window
- Example: Reading papers with LLMs
- Asking for summaries and explanations of complex papers
- Helps understand papers outside one's expertise
- Reading books with LLMs
- Example: The Wealth of Nations by Adam Smith
- Copy-pasting chapters and asking for summaries, explanations
- Increases retention and understanding
- Particularly useful for older texts or specialized fields
Python Interpreter and Programming
- Models can write and execute code
- Used for calculations, data analysis, visualizations
- Model writes code, application executes it, returns results
- Example: Complex multiplication
- Simple calculations done directly by model
- Complex calculations trigger Python interpreter
- Different models have different tool access (some hallucinate answers)
- Advanced Data Analysis in ChatGPT
- Creates visualizations and performs data analysis
- Example: Analyzing OpenAI's valuation over time
- Warning: Models can make implicit assumptions or errors in code
- Cursor app for professional coding
- Integrates with Claude 3.7 Sonnet
- Has context of all files in directory
- Example: Building Tic-Tac-Toe game with confetti effect
- "Vibe coding" - giving high-level commands to the model
Multimodal Capabilities
- Speech/Audio capabilities:
- Input: Converting speech to text
- Mobile apps have built-in functionality
- Desktop requires third-party tools (Super Whisper, Whisper Flow)
- Output: Text-to-speech for reading responses
- True audio processing (Advanced Voice Mode)
- Handles audio natively within model
- Allows for voice character changes, storytelling
- Input: Converting speech to text
- Image capabilities:
- Input: Analyzing uploaded images
- Example: Nutrition label analysis
- Blood test results interpretation
- Mathematical expressions
- Toothpaste ingredient analysis
- Meme explanation
- Output: Image generation with DALL-E
- Can create images based on prompts
- Used for content creation, thumbnails
- Input: Analyzing uploaded images
- Video capabilities:
- Mobile app can analyze video through camera
- Can identify objects, read text, describe scenes
- Video generation tools rapidly evolving (Sora, Gen2, etc.)
- Notebook LM from Google
- Creates custom podcasts from uploaded documents
- Interactive mode allows questions during playback
Quality of Life Features
- Memory feature in ChatGPT
- Retains information about user preferences across conversations
- Creates database of knowledge about the user
- Example: Remembering opinion about peak Hollywood era
- Custom instructions
- Modify how model speaks to you
- Set personal preferences, identity, language learning settings
- Custom GPTs
- Save specific prompts for repeated tasks
- Example: Korean vocabulary extractor
- Converts sentences to flashcard format
- Example: Detailed translator
- Breaks down sentences part by part
- Example: OCR + translation for TV subtitles
- Processes screenshots, extracts text, translates
