AI summary by TwinMind
XGBoost Part 2 (of 4): Classification
StatQuest with Josh Starmer
25m 17s
Introduction and Overview
- XGBoost classification tutorial (Part 2 of series)
- Assumes familiarity with XGBoost regression and gradient boosting classification
- Uses simple training data: 4 drug dosages (green dots = effective, red dots = not effective)
- XGBoost is complex algorithm with many simple, understandable parts
Initial Prediction Setup
- First step: make initial prediction (default = 0.5 probability regardless of regression/classification)
- Default prediction: 50% chance drug is effective for any dosage
- Graph setup: y-axis represents probability drug is effective
- Effective dosages (green) moved to probability = 1, ineffective (red) at probability = 0
- Residuals calculated as differences between observed and predicted values
Similarity Score Formula for Classification
- New formula for classification similarity scores
- Numerator: sum of residuals squared (same as regression)
- Denominator: lambda + sum for each observation of (previous probability × (1 - previous probability))
- Different from regression denominator but closely related (explained in Part 3)
Building the First Tree
- Tree starts as single leaf with all residuals
- Root similarity score = 0 (residuals cancel out in numerator)
- Split threshold: dosage < 15 (average between last two observations)
- Left leaf: 3 residuals (dosage < 15), Right leaf: 1 residual (dosage ≥ 15)
- Lambda = 0 for simplicity (reduces similarity scores when > 0)
Similarity Score Calculations
- Left leaf similarity: 0.33 (two residuals cancel, one remains)
- Right leaf similarity: 1.0
- Gain calculation: 0.33 + 1.0 - 0 = 1.33
- Dosage < 15 threshold provides highest gain
Further Tree Splitting
- Split left branch further: dosage < 5 vs dosage < 10
- Dosage < 5 threshold: gain = 2.66
- Dosage < 10 threshold: gain = 0.66
- Choose dosage < 5 (higher gain)
- Tree limited to 2 levels
Cover and Minimum Leaf Requirements
- Cover = denominator of similarity score - lambda
- For classification: cover = sum of (previous probability × (1 - previous probability))
- For regression: cover = number of residuals in leaf
- Default minimum cover = 1
- Example calculations show leaves would be rejected with default minimum
- Set minimum cover = 0 (min_child_weight = 0) to allow tree building
Tree Pruning
- Prune by comparing gain to gamma parameter
- If (gain - gamma) < 0, prune the branch
- Example: gamma = 2 → no pruning, gamma = 3 → prune branches
- Lambda > 0 reduces similarity scores → lower gains → easier pruning
- Lambda reduces tree sensitivity to individual observations
Output Values for Leaves
- Formula: sum of residuals / (sum of previous probability × (1 - previous probability) + lambda)
- Similar to non-extreme gradient boost formula (except lambda)
- Example calculations with lambda = 0: output values = -2, 2, -2
- Lambda > 0 shrinks output values toward zero (regularization effect)
Making New Predictions
- Convert initial probability (0.5) to log odds = 0
- New prediction = initial log odds + (learning rate × tree output)
- Default learning rate (eta) = 0.3
- Convert log odds back to probability using logistic function
- Example: dosage 2 → log odds = -0.6 → probability = 0.35
Prediction Examples
- Dosage 2: 0.5 → 0.35 (residual reduced)
- Dosage 8: 0.5 → 0.65 (residual reduced)
- All new predictions have smaller residuals (progress toward correct values)
Building Additional Trees
- Build second tree using new residuals
- Previous probabilities now different for each observation
- Similarity score calculations more complex (mixed previous probabilities)
- Continue building trees until residuals very small or max trees reached
Summary of Key Concepts
- Similarity scores and gain determine data splitting
- Pruning using gamma parameter (gain - gamma comparison)
- Lambda regularization: shrinks similarity scores and output values
- Cover metric determines minimum residuals per leaf
- Classification requires log odds conversions and logistic function
- Process continues iteratively building trees on residuals
