AI summary by TwinMind

CatBoost Part 2: Building and Using Trees

StatQuest with Josh Starmer

16m 15s

Introduction & Prerequisites

  • StatQuest tutorial on CatBoost part 2: building and using trees
  • Prerequisites: CatBoost part 1 (ordered target encoding), cosine similarity knowledge
  • Simple dataset example: favorite color predicting height

CatBoost Tree Building Process

  • Each tree creation starts with randomizing training data rows
  • Applies ordered target encoding to discrete columns with >2 options
  • Binary columns (≤2 options) replaced with 1s and 0s

Continuous Target Variable Handling

  • Continuous height values put into discrete bins
  • Example: 2 bins roughly equal size
    • Bin 0: two smallest height values
    • Bin 1: three larger height values
  • Bin numbers used for ordered target encoding (same as categorical case)

Ordered Target Encoding Key Concepts

  • Pretends receiving data sequentially one row at a time
  • Prevents leakage: row's target value doesn't affect same row's encoding
  • More data → more bins, different equation (documented on CatBoost website)

Initial Model Setup

  • Model predictions initialized to zero for all rows
  • Initial residuals calculated: observed heights - predicted heights
  • Example: first row residual = 1.81 - 0 = 1.81

Tree Building: Threshold Selection

  • Sort favorite color values low to high
  • Identify potential thresholds between values
  • Example thresholds: 0.04 (between 0.025 and 0.05), 0.29 (between 0.05 and 0.525)
  • CatBoost limits max thresholds by binning close values

Stump Building Process (Simplified Tree)

  • Initialize leaf outputs to zero
  • Run each row down tree sequentially:
    • Place residual in appropriate leaf
    • Track current output value
    • Update leaf output to average of residuals in leaf
  • Example for threshold 0.04:
    • Row 1: goes right (0.05 > 0.04), output updates to 1.81
    • Row 2: goes right, output updates to 1.69 (average)
    • Row 3: goes left, left leaf updates to 1.77
    • Rows 4-5: go right, final right leaf output 1.58

Tree Evaluation: Cosine Similarity

  • Quantifies prediction quality using cosine similarity
  • Between leaf output column and residuals column
  • Threshold 0.04: cosine similarity = 0.71
  • Threshold 0.29: cosine similarity = 0.79
  • Higher cosine similarity = better threshold
  • Selected threshold: 0.29 (higher score)

Data Limitations & Practical Considerations

  • Small dataset (5 rows) uses all data for cosine similarity
  • Includes leaf outputs = 0 (initialization values not based on data)
  • In practice with more data: CatBoost ignores first bunch of rows in calculation

Prediction Updates & Next Tree

  • Update predictions: add (learning rate × leaf output)
  • Learning rate example: 0.1
  • New prediction = old prediction + (0.1 × leaf output)
  • Calculate new residuals: observed - new predictions
  • Progress: predictions improve but still not good

Sequential Data Processing Philosophy

  • Key CatBoost concept: treats data as arriving sequentially
  • Row's residual not part of same row's leaf output calculation
  • Avoids leakage in prediction calculation
  • Similar principle to ordered target encoding

Second Tree Building

  • Update residuals with new predictions
  • Replace encoded values with original color names
  • Re-randomize rows
  • Reuse bin numbers for new ordered target encoding
  • Identify new thresholds, build tree with same process

Making Predictions on New Data

  • Encode new data using original training data bins
  • Run through all trees
  • Sum: (learning rate × leaf outputs) from all trees
  • Example prediction: 0.3 (terrible but demonstrates process)

Symmetric/Oblivious Decision Trees

  • CatBoost builds symmetric decision trees for larger trees
  • Same threshold used for all nodes at same level
  • Example: both second-level nodes use "age < 12"

Reasons for Symmetric Trees

  1. Weaker Learners: Makes trees worse at predictions
    • Fits gradient boosting philosophy of combining weak learners
  2. Faster Predictions:
    • No need to track which questions to ask
    • Same questions asked regardless of path
    • Single vector operation possible
    • More efficient than normal trees

CatBoost Key Differentiators

  • Sequential data treatment to avoid leakage
  • Applies to both target encoding and tree output calculations
  • Symmetric trees for weakness and speed
  • Combines multiple weak learners effectively

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome