AI summary by TwinMind

Live Stream - More details about Target Encoding/AMA/Silly Songs

StatQuest with Josh Starmer

51m 17s

Introduction and Setup

  • Josh Starmer hosting StatQuest live stream
  • Woke up at 1am, been awake 11+ hours, apologetic about being sleepy
  • Apologized for not practicing Tabla as promised in previous stream
  • Mentioned having Tambora instrument, considering friend collaboration for drone background
  • Encouraged questions in chat, will address at end of session

Basic Encoding Strategies Review

  • Dataset: favorite color (blue/red/green) + height to predict "loves Troll 2"
  • One-hot encoding works for few discrete values but problematic with many options
    • Example: 41,683 postal codes in US would create 41,683 new columns
  • Target encoding as alternative: replace each option with weighted mean of target

Standard Target Encoding Process

  • Weighted mean formula: (n × option_mean + m × overall_mean) / (n + m)
  • Blue example: (3 × 1/3 + 2 × 3/7) / (3 + 2) = 0.37
  • Red example: (2 × 0/2 + 2 × 3/7) / (2 + 2) = 0.29
  • Green example: (2 × 2/2 + 2 × 3/7) / (2 + 2) = 0.57
  • M=2 hyperparameter: need ≥3 rows before option mean becomes more important than overall mean

Leakage Problem

  • Using target values to modify feature values creates leakage
  • Results in overfitted models: great on training data, poor on testing data
  • Relationship established between target and encoded features

K-Fold Target Encoding Solution

  • Split data into K equal-sized subsets
  • Two-fold example: subset A (4 rows), subset B (3 rows)
  • Encode subset A using only target values from subset B
  • Encode subset B using only target values from subset A
  • Blue in subset A: 0.22, Blue in subset B: 0.5
  • Different values for same category across subsets is acceptable
  • Leave-one-out encoding: K=7 (number of total rows)
  • Alternative: K=5 commonly used with large datasets

CatBoost Ordered Target Encoding

  • Treats samples as sequential input
  • Only uses data from preceding rows for encoding
  • Different equation: (count_in_class + prior) / (total_count + 1)
  • Prior typically 0.05 (from documentation example)
  • First occurrences get prior value (0.05)
  • Subsequent occurrences use accumulated statistics from previous rows
  • Blue sequence: 0.05 → 0.525 → 0.35
  • Green sequence: 0.05 → 0.525 → 0.67

CatBoost Justification Critique

  • Authors claim leave-one-out causes leakage with single option
  • Josh argues this justification doesn't make sense
  • When only 1-2 options exist, everyone uses one-hot encoding
  • CatBoost itself uses one-hot encoding for few categories
  • Many practitioners successfully use leave-one-out encoding
  • Despite weak justification, CatBoost works well in practice

Machine Learning Philosophy

  • Unlike pure mathematics, ML focuses on results over perfect logic
  • Sometimes requires "getting messy" to achieve good performance
  • Pragmatic approach: if it works well, it's valuable

Q&A Session Highlights

  • Inference on new data: use all training data to encode (no leakage concern)
  • Regression targets: same process, just calculate means of continuous values
  • Book availability: Amazon for US/Europe, working on India/Australia/Brazil printing
  • Jobs: StatQuest, Lightning AI educator, book writing
  • Leakage definition: when prediction target influences feature encoding
  • Feature engineering: normalization, ratios, combining features
  • Ordinal data: often treated as numeric without encoding needed

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome