AI summary by TwinMind
CatBoost Part 2: Building and Using Trees
StatQuest with Josh Starmer
16m 15s
Introduction & Prerequisites
- StatQuest tutorial on CatBoost part 2: building and using trees
- Prerequisites: CatBoost part 1 (ordered target encoding), cosine similarity knowledge
- Simple dataset example: favorite color predicting height
CatBoost Tree Building Process
- Each tree creation starts with randomizing training data rows
- Applies ordered target encoding to discrete columns with >2 options
- Binary columns (≤2 options) replaced with 1s and 0s
Continuous Target Variable Handling
- Continuous height values put into discrete bins
- Example: 2 bins roughly equal size
- Bin 0: two smallest height values
- Bin 1: three larger height values
- Bin numbers used for ordered target encoding (same as categorical case)
Ordered Target Encoding Key Concepts
- Pretends receiving data sequentially one row at a time
- Prevents leakage: row's target value doesn't affect same row's encoding
- More data → more bins, different equation (documented on CatBoost website)
Initial Model Setup
- Model predictions initialized to zero for all rows
- Initial residuals calculated: observed heights - predicted heights
- Example: first row residual = 1.81 - 0 = 1.81
Tree Building: Threshold Selection
- Sort favorite color values low to high
- Identify potential thresholds between values
- Example thresholds: 0.04 (between 0.025 and 0.05), 0.29 (between 0.05 and 0.525)
- CatBoost limits max thresholds by binning close values
Stump Building Process (Simplified Tree)
- Initialize leaf outputs to zero
- Run each row down tree sequentially:
- Place residual in appropriate leaf
- Track current output value
- Update leaf output to average of residuals in leaf
- Example for threshold 0.04:
- Row 1: goes right (0.05 > 0.04), output updates to 1.81
- Row 2: goes right, output updates to 1.69 (average)
- Row 3: goes left, left leaf updates to 1.77
- Rows 4-5: go right, final right leaf output 1.58
Tree Evaluation: Cosine Similarity
- Quantifies prediction quality using cosine similarity
- Between leaf output column and residuals column
- Threshold 0.04: cosine similarity = 0.71
- Threshold 0.29: cosine similarity = 0.79
- Higher cosine similarity = better threshold
- Selected threshold: 0.29 (higher score)
Data Limitations & Practical Considerations
- Small dataset (5 rows) uses all data for cosine similarity
- Includes leaf outputs = 0 (initialization values not based on data)
- In practice with more data: CatBoost ignores first bunch of rows in calculation
Prediction Updates & Next Tree
- Update predictions: add (learning rate × leaf output)
- Learning rate example: 0.1
- New prediction = old prediction + (0.1 × leaf output)
- Calculate new residuals: observed - new predictions
- Progress: predictions improve but still not good
Sequential Data Processing Philosophy
- Key CatBoost concept: treats data as arriving sequentially
- Row's residual not part of same row's leaf output calculation
- Avoids leakage in prediction calculation
- Similar principle to ordered target encoding
Second Tree Building
- Update residuals with new predictions
- Replace encoded values with original color names
- Re-randomize rows
- Reuse bin numbers for new ordered target encoding
- Identify new thresholds, build tree with same process
Making Predictions on New Data
- Encode new data using original training data bins
- Run through all trees
- Sum: (learning rate × leaf outputs) from all trees
- Example prediction: 0.3 (terrible but demonstrates process)
Symmetric/Oblivious Decision Trees
- CatBoost builds symmetric decision trees for larger trees
- Same threshold used for all nodes at same level
- Example: both second-level nodes use "age < 12"
Reasons for Symmetric Trees
- Weaker Learners: Makes trees worse at predictions
- Fits gradient boosting philosophy of combining weak learners
- Faster Predictions:
- No need to track which questions to ask
- Same questions asked regardless of path
- Single vector operation possible
- More efficient than normal trees
CatBoost Key Differentiators
- Sequential data treatment to avoid leakage
- Applies to both target encoding and tree output calculations
- Symmetric trees for weakness and speed
- Combines multiple weak learners effectively
