AI summary by TwinMind
XGBoost Part 1 (of 4): Regression
StatQuest with Josh Starmer
25m 46s
Introduction and Prerequisites
- XGBoost = Extreme Gradient Boost
- Presenter: Josh Stormer from StatQuest
- Assumes familiarity with:
- Gradient boost regression concepts
- Regularization main ideas
- XGBoost = big machine learning algorithm with many simple parts
- Series structure: Part 1 (regression), Part 2 (classification), Part 3 (mathematical details)
Training Data Setup
- Simple dataset used for demonstration (XGBoost designed for large/complex datasets)
- X-axis: drug dosages
- Y-axis: drug effectiveness measurements
- Two observations: large positive values (drug helpful)
- Two observations: large negative values (drug harmful)
Initial Prediction Process
- First step: make initial prediction
- Default initial prediction = 0.5 (regardless of regression/classification)
- Corresponds to horizontal line on graph
- Calculate residuals = observed - predicted values
- Shows quality of initial prediction
XGBoost Tree Construction vs Standard Gradient Boost
- Standard gradient boost: uses regular off-the-shelf regression trees
- XGBoost: uses unique regression trees ("XGBoost trees")
- Focus on most common way to build XGBoost trees for regression
Similarity Score Calculation
- Each tree starts as single leaf with all residuals
- Calculate quality/similarity score for residuals
- Formula: Similarity = (sum of residuals)² / (number of residuals + λ)
- λ (lambda) = regularization parameter
- Initially set λ = 0 for examples
- Example calculation: residuals 7.5, -7.5, 6.5, -10.5
- Sum = -4 (cancellation occurs)
- Similarity = (-4)² / 4 = 4
Tree Splitting Process
- Test different thresholds to find best split
- Threshold 1: dosage < 15
- Left leaf: 1 residual (-10.5), similarity = 110.25
- Right leaf: 3 residuals, similarity = 14.08
- Threshold 2: dosage < 22.5, gain = 4
- Threshold 3: dosage < 30, gain = 56.33
- Choose threshold with largest gain
Gain Calculation
- Formula: Gain = Left similarity + Right similarity - Root similarity
- Threshold dosage < 15: Gain = 110.25 + 14.08 - 4 = 120.33
- Best threshold selected based on highest gain
- Continue splitting process for remaining nodes
Tree Pruning with Gamma
- Pruning based on gain values
- Gamma (γ) = tree complexity parameter (user-defined number)
- Pruning rule: If (Gain - γ) < 0, remove branch
- Example with γ = 130: keep branches (positive difference)
- Example with γ = 150: remove branches (negative difference)
- Prevents overfitting through aggressive pruning
Regularization Parameter Lambda Effects
- When λ = 1 (vs λ = 0):
- Root similarity: 3.2 (vs 4.0) - 20% decrease
- Left leaf similarity: 55.12 (vs 110.25) - 50% decrease
- Right leaf similarity: 10.56 (vs 14.08) - 25% decrease
- Pattern: Decrease inversely proportional to number of residuals
- Fewer residuals = larger decrease in similarity score
- Results in smaller gain values, easier pruning
- Can create negative gains even with γ = 0
Output Value Calculation
- Formula: Output = (sum of residuals) / (number of residuals + λ)
- Different from similarity score (no squaring of sum)
- When λ = 0: output = average of residuals
- When λ > 0: reduces individual observation impact
- Example calculations:
- Left leaf: -10.5/1 = -10.5 (λ=0) vs -5.25 (λ=1)
- Middle leaf: 7 (average of residuals)
- Right leaf: -7.5
Making New Predictions
- Formula: New prediction = Initial prediction + η × Tree output
- η (eta) = learning rate, default = 0.3
- Example: dosage=10, new prediction = 0.5 + 0.3×(-10.5) = -2.65
- New residuals smaller than original (progress in right direction)
- Repeat process: build tree → make predictions → calculate residuals
Iterative Process
- Continue building trees on new residuals
- Stop when:
- Residuals become very small
- Maximum number of trees reached
- Each iteration takes small step toward better predictions
Key Parameters Summary
- λ (lambda): regularization parameter
- Controls similarity scores and output values
- Higher λ = more pruning, smaller outputs
- Reduces prediction sensitivity to individual observations
- γ (gamma): tree complexity parameter
- Controls pruning threshold
- Higher γ = more aggressive pruning
- η (eta): learning rate, default = 0.3
- Controls step size in predictions
- Tree depth: default maximum = 6 levels
