Introduction and Prerequisites

• Josh Starmer from StatQuest tutorial on gradient boost algorithmic details for regression
• Assumes viewers watched "gradient boost part one regression main ideas"
• Recommends understanding gradient descent (optional but helpful)
• Will walk through original gradient boost algorithm step-by-step

Training Dataset Setup

• Super simple dataset with 3 people
• Features: height measurements, favorite color, gender
• Target: weight
• n = 3 samples total
• Data notation: x_i (feature rows), y_i (weights), i goes from 1 to n

Loss Function Selection

• Uses differentiable loss function: 1/2 * (observed - predicted)²
• Same as linear regression loss function (without 1/2 multiplier)
• 1/2 factor chosen for mathematical convenience
• When differentiated: derivative = -(observed - predicted) = negative residual
• Makes math easier since gradient boost uses derivatives frequently
• Residuals = differences between observed and predicted weights

Step 1: Model Initialization

• Initialize model with constant value F₀(x)
• Constant determined by arg min over γ of sum of loss functions
• Minimize sum of squared residuals/2 across all samples
• Solution: γ = average of observed weights
• F₀(x) = 73.3 (average weight)
• Initial prediction: leaf predicting all samples weigh 73.3

Step 2: Tree Building Loop (M iterations)

• Generic: build M trees (typically M = 100 in practice)
• Little m = individual tree index (1 to M)
• Starting with m = 1 (first tree)

Step 2A: Calculate Pseudo-Residuals

• r_{i,m} = negative gradient of loss function
• r_{i,m} = observed - predicted = y_i - F_{m-1}(x_i)
• For m=1: use F₀(x) = 73.3 as previous prediction
• Calculate residuals for all samples:
  • r_{1,1} = 88 - 73.3 = 14.7
  • r_{2,1} = 76 - 73.3 = 2.7
  • r_{3,1} = 66 - 73.3 = -7.3
• Called "pseudo-residuals" because with other loss functions wouldn't be exact residuals

Step 2B: Fit Regression Tree to Residuals

• Build regression tree using height, color, gender to predict residuals (not weights)
• Creates tree/stump with decision rule
• Example: if height ≤ 1.55 → left leaf, else → right leaf
• Sample x₃ → left leaf (residual -7.3)
• Samples x₁, x₂ → right leaf (residuals 14.7, 2.7)
• Create terminal regions R_{j,m} (leaves labeled R₁,₁ and R₂,₁)

Step 2C: Calculate Output Values for Each Leaf

• For each leaf, compute γ_{j,m} that minimizes sum of loss functions
• Only include samples that end up in that specific leaf
• Left leaf R₁,₁ (only x₃):
  • Minimize: 1/2 * (66 - (73.3 + γ))²
  • Solution: γ₁,₁ = -7.3
• Right leaf R₂,₁ (x₁ and x₂):
  • Minimize sum over both samples
  • Solution: γ₂,₁ = 8.7 (average of residuals in leaf)
• Output values always equal average of residuals in each leaf

Step 2D: Make New Predictions

• F_m(x) = F_{m-1}(x) + ν * Σ γ{j,m} * I(x ∈ R{j,m})
• ν = learning rate (set to 0.1)
• Small learning rate reduces each tree's effect, improves long-term accuracy
• New predictions F₁(x):
  • x₁: 73.3 + 0.1 * 8.7 = 74.2
  • x₂: 73.3 + 0.1 * 8.7 = 74.2
  • x₃: 73.3 + 0.1 * (-7.3) = 71.6
• All predictions improved (closer to observed weights)

Iteration Process

• Repeat Step 2 for m = 2, 3, ..., M
• Each iteration:
  • Calculate new residuals using latest predictions
  • Build new tree to predict residuals
  • Calculate leaf output values
  • Update predictions
• Assuming M = 2 for demonstration (practice uses M ≥ 100)

Step 3: Final Output

• Final model: F_M(x) = F₂(x) after M iterations
• For new data prediction:
  • Start with initial leaf value (73.3)
  • Add contributions from all trees scaled by learning rate
  • Example prediction: 73.3 + 0.1(-7.3) + 0.1(-15.6) = 70 kg

Important Notes

• Tutorial uses stumps due to small dataset (only 3 samples)
• Gradient boost typically uses larger trees in practice
• Next tutorial will cover gradient boost for classification