AI summary by TwinMind
How to Prune Regression Trees, Clearly Explained!!!
StatQuest with Josh Starmer
16m 15s
Introduction and Prerequisites
- Host: Josh Starmer, StatQuest
- Topic: Cost complexity pruning (aka weakest link pruning) for regression trees
- Prerequisites: familiarity with regression trees and cross-validation
- Purpose: prevent overfitting to improve performance on testing data
Overfitting Problem Example
- Dataset: drug dosage (x-axis) vs effectiveness (y-axis)
- Pattern: low/high dosages ineffective, medium dosages very effective, moderately high dosages moderately effective
- Full regression tree fits training data well (each leaf = average effectiveness from observation clusters)
- Testing data analysis reveals overfitting:
- Some predictions close to actual values (small residuals)
- Four observations with 100% effectiveness appear as outliers
- Large residuals indicate overfitting to training data
Pruning Solution Concept
- Remove leaves and replace splits with leaf averaging larger observation groups
- Example: observations between 14.5-29 go to single leaf on far right
- Trade-off: worse fit to training data but better performance on testing data
- Progressive pruning possible: can remove multiple leaf pairs sequentially
Cost Complexity Pruning Methodology
Step 1: Calculate Sum of Squared Residuals (SSR)
- Original full-sized tree SSR breakdown:
- Dosages < 14.5: SSR = 320.8
- Dosages ≥ 29: SSR = 75
- Dosages ≥ 23 and < 29: SSR = 148.8
- Dosages ≥ 14.5 and < 23.5: SSR = 0
- Total SSR = 543.8
- Pruned trees SSR calculations:
- 3-leaf subtree: SSR = 5,494.8
- 2-leaf subtree: SSR = 19,243.7
- 1-leaf subtree: SSR = 28,897.2
Step 2: Tree Scoring Formula
- Tree Score = SSR + α × T
- Where: α = tuning parameter, T = number of leaves
- Tree complexity penalty = α × T (compensates for leaf count differences)
Step 3: Example with α = 10,000
- Full tree (4 leaves): Score = 543.8 + 10,000 × 4 = 40,543.8
- 3-leaf subtree: Score = 5,494.8 + 10,000 × 3 = 35,494.8 (lowest score)
- 2-leaf subtree: Score = 19,243.7 + 10,000 × 2 = 39,243.7
- 1-leaf subtree: Score = 28,897.2 + 10,000 × 1 = 38,897.2
Step 4: Alpha Impact Analysis
- With α = 22,000: 1-leaf subtree has lowest score
- Alpha value determines optimal tree selection
Complete Pruning Process
Phase 1: Build Full Tree and Find Alpha Values
- Use all data to build full-sized regression tree
- α = 0: full tree has lowest score (no complexity penalty)
- Incrementally increase α to find pruning thresholds:
- α = 10,000: prune to 3-leaf subtree
- α = 15,000: prune to 2-leaf subtree
- α = 22,000: prune to 1-leaf subtree
- Creates sequence from full tree to single leaf
Phase 2: Cross-Validation Process
- Split full dataset into training/testing sets
- Using training data only, build tree sequence with found α values
- Calculate SSR for each tree using testing data
- Example results: tree with α = 10,000 had smallest testing SSR
- Repeat with new training/testing splits
- Continue until 10-fold cross-validation complete
Phase 3: Final Selection
- Identify α value with lowest average testing SSR across all folds
- Example: α = 10,000 produced optimal results on average
- Return to original full-data trees, select tree corresponding to optimal α
- This becomes final pruned regression tree
