AI summary by TwinMind

Luis Serrano + Jay Alammar + Josh Starmer Q&A Livestream!!!

StatQuest with Josh Starmer

59m 55s

Introduction and Background

  • Live stream Q&A featuring Josh Starmer (StatQuest), Luis Serrano, and Jay Alammar
  • Participants joining from Canada, North Carolina, Saudi Arabia, Australia, India, Bolivia, Cyprus, Montreal
  • Discussion format: pre-submitted questions plus live chat interaction

Content Creator Origins and Styles

  • Josh Starmer (StatQuest):
    • Originally created content for genetics lab co-workers at UNC Chapel Hill
    • Friday morning statistics presentations for mouse genetics researchers
    • Target audience: smart people from non-mathematical backgrounds
    • Style developed to make statistics exciting for people who hated it
    • Created characters like "Squatch and Norm," catchphrases like "bam" and "double bam"
  • Luis Serrano:
    • Started at Udacity making videos for teammates 8 years ago
    • Goal: accessible to beginners while still valuable for advanced learners
    • Running gag with mountain names (Mount Errorist, Mount Killerror, etc.)
    • Ended mountain theme after creating "Eyjafjallajökull" variation in Iceland
  • Jay Alammar:
    • Started with blogging, highly visual content
    • Influenced by watching hundreds of hours of StatQuest and Luis's videos
    • Transitioned to Apple Keynote after lunch with Luis
    • Explores multiple platforms: YouTube, LinkedIn, TikTok, Instagram Reels, YouTube Shorts

Technical Tools and Methods

  • All three creators use Apple Keynote for presentations
  • Josh learned Keynote techniques from Jay's tutorial videos
  • Emphasis on visual storytelling and step-by-step explanations

Time Series and Machine Learning Applications

  • XGBoost: Effective for time series applications
  • Recurrent Neural Networks: Traditional choice for time series
  • Hidden Markov Models: Still relevant for certain applications
  • 1D Convolutional Networks: Good for time series data
  • ARIMA: Combines multiple approaches, used in many background tools
  • Transformers vs RNNs: If solvable with RNN, likely better with Transformer
  • State Space Models (SSMs): Promising hybrid approach, showing potential but not yet surpassing Transformers in text generation

Generative AI Beyond NLP

  • Multiple Modalities: Video generation, image generation, audio generation
  • Understanding vs Generation: Representation learning equally important
  • Computer Vision Applications: Stable diffusion, diffusion models
  • Encoder-Decoder Architecture: Understanding component feeds generation component
  • Modern Approach: Can start directly with Transformers without traditional NLP background
  • Historical Context: Pre-Transformer NLP focused on parsing, hierarchical structures, linguistic rules
  • Paradigm Shift: Transformers use data averaging approach vs. structured linguistic analysis

Learning Recommendations and Roadmap

  • Starting Point: Begin with LLMs and prompting (ChatGPT, Claude, etc.)
  • Two-Track Approach:
    • Theory: Understanding how components work
    • Practice: Hands-on experimentation with datasets
  • Learning Philosophy: "Meet in the middle" - theoretical understanding + practical application
  • Data Selection: Use interesting datasets (sports, video games, personal interests)
  • Learn in Public: Document and share learning process
  • Multiple Sources: Try different explanations and teaching styles
  • Collaboration: Join Discord servers, open science groups (Cohere for AI, Hugging Face projects)
  • Contribution: Participate in open-source projects, label data, take on tasks

Jay Alammar's Upcoming Book

  • Title: "Hands-On Large Language Models"
  • Publisher: O'Reilly
  • Release: September (approximately 1.5 months away)
  • Length: ~400 pages with ~300 original images
  • Content: Transformers explanation, applications, fine-tuning, under-the-hood mechanics
  • Chapter 3: Updated version of "The Illustrated Transformer" (7 years later)
  • Focus: Modern text generation Transformers with practical applications
  • Examples: Document clustering, summarization, labeling - practical use cases beyond basic prompting

Career Guidance and Job Search

  • Early Application Strategy: Apply for internships while still learning
  • Interview Practice: Use interviews as learning opportunities
  • Portfolio Development: Kaggle projects acceptable as professional examples
  • Internship Opportunities: Available at both large and small companies, including Cohere
  • GitHub Presence: Well-documented, annotated project repositories
  • Learn in Public Benefits:
    • 70% of jobs filled before advertising
    • Network-based hiring common
    • Establishes expertise reputation
    • Creates opportunities in hidden job market
  • Paper Explanation Strategy: Write explanations of research papers, send to authors for feedback
  • Networking: Attend events, make connections, build relationships
  • Referral Importance: Most jobs obtained through referrals and network connections

Graph Neural Networks Discussion

  • Data Structure Fit: Different neural networks for different data types
    • Regular NNs: Row/column independence
    • CNNs: Neighbor correlation (images)
    • RNNs: Sequential data
    • GNNs: Graph-structured data
  • Applications: Many datasets naturally graph-structured
  • Related Technologies: Graph databases
  • Current Status: Active research area, not deeply explored by panel participants

Retrieval Augmented Generation (RAG)

  • Basic Concept: Two-step process - search first, then generate answer
  • Problem Solved: LLMs not designed as search engines
  • Basic RAG: Question + relevant context → better answers
  • Advanced Improvements:
    • Hybrid Search: Keyword search + embedding search
    • Reranking: Improve top 10-100 results
    • Query Rewriting: Extract keywords from long questions
    • Multi-Query RAG: Handle questions requiring multiple searches
    • Multi-Hop RAG: Sequential searches building on previous results
    • Grounded Generation: Answer based strictly on provided context
  • Evolution to Agents: LLMs performing reasoning, using multiple tools, taking actions
  • Limitations: Complex comparative queries across many documents still challenging
  • Long Context Factor: Context window size affects capability

Fine-Tuning vs RAG

  • RAG Preference: Better for grounding LLMs to specific datasets or factual information
  • Fine-Tuning Applications:
    • Domain adaptation
    • Specific output formatting
    • Last resort after prompt engineering and RAG
  • Analogy: Fine-tuning = studying for masters degree; RAG = carrying reference book/phone
  • Data Currency: RAG provides instant access to updated information
  • Combination Approach: Both methods together plus good prompting for optimal results
  • Trust and High-Stakes: Human-in-the-loop required for critical decisions (medical, hiring)

Three-Stage LLM Training Process

  • Stage 1 - Base Training: 90-95% of resources, next-token prediction, requires high-quality tokens without errors
  • Stage 2 - Supervised Fine-Tuning/Instruction Tuning: Teaching request fulfillment vs. continuation prediction
  • Stage 3 - Preference Tuning/Alignment: Using methods like DPO (Direct Preference Optimization), comparing good vs. worse completions

Code Generation and Copilot Applications

  • Base Training Requirements: Extensive code exposure in training data
  • Interaction Methods:
    • Autocomplete/generation from function definitions
    • Repository question answering
  • Data Structure: Instruction-completion pairs for supervised fine-tuning phase
  • Tool Integration: Models embedded in development environments (GitHub Copilot, Replit)

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome