## Introduction and Prerequisites - Regression trees tutorial by Josh Starman from StatQuest - Assumes familiarity with: - Bias-variance tradeoff in machine learning - Basic decision tree concepts - Basic regression concepts ## Drug Dosage Example - Linear vs Non-Linear Data - Hypothetical drug study for common cold cure - Clinical trial measuring effectiveness vs dosage - **Linear case**: Higher dose = more effective, straight line fits well - 27mg dose → 62% effectiveness prediction - **Non-linear case**: Complex relationship - Low dosages: not effective - Moderate dosages: work really well - Higher dosages: ~50% effectiveness - High dosages: not effective at all - Straight line prediction: 20mg dose → 45% (incorrect, should be 100%) ## Regression Trees vs Classification Trees - **Regression trees**: leaves contain numeric values - **Classification trees**: leaves contain true/false or discrete categories - Better suited for non-linear relationships than straight lines ## Single Predictor Regression Tree Structure - **Root question**: dosage < 14.5? - **Left branch** (dosage < 14.5): - 6 observations in training data - Average effectiveness: 4.2% - Prediction: 4.2% - **Right branch splits further**: - If dosage ≥ 29: 4 observations, average 2.5% → prediction 2.5% - If 23.5 ≤ dosage < 29: 5 observations, average 52.8% → prediction 52.8% - If 14.5 ≤ dosage < 23.5: 4 observations, average 100% → prediction 100% ## Multi-Predictor Capability - Trees easily handle 3+ predictors (dosage, age, sex) - Example prediction for patient: - Age > 50? No → right branch - Dosage ≥ 29? No → right branch - Female? Yes → left branch - Prediction: 100% (actual: 98%) - Graphs become impossible with 3+ dimensions, trees remain manageable ## Building Regression Trees from Scratch ### Threshold Selection Process - Start with smallest dosages, calculate average (3) - Build simple tree: dosage < 3 - Left leaf: 0% (1 observation) - Right leaf: 38.8% (all other observations) - Calculate sum of squared residuals for each threshold - Test all possible thresholds between consecutive dosage values - **Optimal threshold**: dosage < 14.5 (smallest sum of squared residuals) ### Sum of Squared Residuals Calculation - For each observation: (observed - predicted)² - Sum all squared residuals - Example with dosage < 3 threshold: 27,468.5 - Dosage < 14.5 had lowest sum of squared residuals ## Tree Building Algorithm ### Left Side Processing (dosage < 14.5) - 6 observations total - Could theoretically split further - **Overfitting problem**: perfect predictions on training data - Individual predictions: 20% → 20%, 5% → 5%, four 0% → 0% - Zero bias but potentially large variance ### Preventing Overfitting - **Minimum observations rule**: typically 20, example uses 7 - Since left side has only 6 observations < 7, make it a leaf - Output: average of 6 observations = 4.2% ### Right Side Processing (dosage ≥ 14.5) - 13 observations > 7 minimum → can split - Find optimal threshold using sum of squared residuals - Split at dosage = 29 - **Right sub-branch** (dosage ≥ 29): 4 observations < 7 → leaf with 2.5% - **Left sub-branch** (14.5 ≤ dosage < 29): 9 observations → can split again - Final split at dosage = 23.5 - Final leaves: - 14.5 ≤ dosage < 23.5: 100% - 23.5 ≤ dosage < 29: 52.8% ## Multi-Predictor Tree Building ### Candidate Selection Process - **Step 1**: Find best dosage threshold → candidate for root - **Step 2**: Find best age threshold → candidate for root - **Step 3**: Find best sex threshold (only one possible) → candidate for root - **Step 4**: Compare sum of squared residuals for all candidates - **Winner**: age > 50 (lowest sum of squared residuals) becomes root ### Tree Growth Process - Continue same process for each branch - Compare optimal thresholds across all predictors - Stop when node has < minimum observations (7 in example, typically 20) - Each leaf outputs average value for observations in that cluster ## Key Algorithm Summary - **Objective**: Minimize sum of squared residuals at each split - **Process**: Try all possible thresholds for each predictor - **Selection**: Choose threshold/predictor combination with lowest sum of squared residuals - **Stopping criteria**: Node has fewer than minimum observations - **Output**: Average of observations in each leaf

Summary by TwinMind · 22m 33s

Regression Trees, Clearly Explained!!!

Regression Trees, Clearly Explained!!!

StatQuest with Josh Starmer