AI summary by TwinMind
Regression Trees, Clearly Explained!!!
StatQuest with Josh Starmer
22m 33s
Introduction and Prerequisites
- Regression trees tutorial by Josh Starman from StatQuest
- Assumes familiarity with:
- Bias-variance tradeoff in machine learning
- Basic decision tree concepts
- Basic regression concepts
Drug Dosage Example - Linear vs Non-Linear Data
- Hypothetical drug study for common cold cure
- Clinical trial measuring effectiveness vs dosage
- Linear case: Higher dose = more effective, straight line fits well
- 27mg dose → 62% effectiveness prediction
- Non-linear case: Complex relationship
- Low dosages: not effective
- Moderate dosages: work really well
- Higher dosages: ~50% effectiveness
- High dosages: not effective at all
- Straight line prediction: 20mg dose → 45% (incorrect, should be 100%)
Regression Trees vs Classification Trees
- Regression trees: leaves contain numeric values
- Classification trees: leaves contain true/false or discrete categories
- Better suited for non-linear relationships than straight lines
Single Predictor Regression Tree Structure
- Root question: dosage < 14.5?
- Left branch (dosage < 14.5):
- 6 observations in training data
- Average effectiveness: 4.2%
- Prediction: 4.2%
- Right branch splits further:
- If dosage ≥ 29: 4 observations, average 2.5% → prediction 2.5%
- If 23.5 ≤ dosage < 29: 5 observations, average 52.8% → prediction 52.8%
- If 14.5 ≤ dosage < 23.5: 4 observations, average 100% → prediction 100%
Multi-Predictor Capability
- Trees easily handle 3+ predictors (dosage, age, sex)
- Example prediction for patient:
- Age > 50? No → right branch
- Dosage ≥ 29? No → right branch
- Female? Yes → left branch
- Prediction: 100% (actual: 98%)
- Graphs become impossible with 3+ dimensions, trees remain manageable
Building Regression Trees from Scratch
Threshold Selection Process
- Start with smallest dosages, calculate average (3)
- Build simple tree: dosage < 3
- Left leaf: 0% (1 observation)
- Right leaf: 38.8% (all other observations)
- Calculate sum of squared residuals for each threshold
- Test all possible thresholds between consecutive dosage values
- Optimal threshold: dosage < 14.5 (smallest sum of squared residuals)
Sum of Squared Residuals Calculation
- For each observation: (observed - predicted)²
- Sum all squared residuals
- Example with dosage < 3 threshold: 27,468.5
- Dosage < 14.5 had lowest sum of squared residuals
Tree Building Algorithm
Left Side Processing (dosage < 14.5)
- 6 observations total
- Could theoretically split further
- Overfitting problem: perfect predictions on training data
- Individual predictions: 20% → 20%, 5% → 5%, four 0% → 0%
- Zero bias but potentially large variance
Preventing Overfitting
- Minimum observations rule: typically 20, example uses 7
- Since left side has only 6 observations < 7, make it a leaf
- Output: average of 6 observations = 4.2%
Right Side Processing (dosage ≥ 14.5)
- 13 observations > 7 minimum → can split
- Find optimal threshold using sum of squared residuals
- Split at dosage = 29
- Right sub-branch (dosage ≥ 29): 4 observations < 7 → leaf with 2.5%
- Left sub-branch (14.5 ≤ dosage < 29): 9 observations → can split again
- Final split at dosage = 23.5
- Final leaves:
- 14.5 ≤ dosage < 23.5: 100%
- 23.5 ≤ dosage < 29: 52.8%
Multi-Predictor Tree Building
Candidate Selection Process
- Step 1: Find best dosage threshold → candidate for root
- Step 2: Find best age threshold → candidate for root
- Step 3: Find best sex threshold (only one possible) → candidate for root
- Step 4: Compare sum of squared residuals for all candidates
- Winner: age > 50 (lowest sum of squared residuals) becomes root
Tree Growth Process
- Continue same process for each branch
- Compare optimal thresholds across all predictors
- Stop when node has < minimum observations (7 in example, typically 20)
- Each leaf outputs average value for observations in that cluster
Key Algorithm Summary
- Objective: Minimize sum of squared residuals at each split
- Process: Try all possible thresholds for each predictor
- Selection: Choose threshold/predictor combination with lowest sum of squared residuals
- Stopping criteria: Node has fewer than minimum observations
- Output: Average of observations in each leaf
