AI summary by TwinMind
StatQuest: Random Forests Part 2: Missing data and clustering
StatQuest with Josh Starmer
11m 53s
Introduction and Dataset Overview
- Josh Stommer presenting StatQuest on Random Forests Part 2
- Focus: missing data handling and sample clustering (presenter's favorite aspect)
- Dataset: 4 patients with some missing data for patient #4
- Two types of missing data scenarios covered
Missing Data Types in Random Forests
- Type 1: Missing data in original dataset used to create random forest
- Type 2: Missing data in new sample requiring categorization
- Starting with Type 1 scenario
Initial Guess Method for Missing Values
- Patient #4 missing: blocked arteries status and weight
- Patient #4 confirmed no heart disease
- Categorical data (blocked arteries): Use most common value from samples with same outcome
- Among no heart disease patients: "no" appears 2/2 times
- Initial guess: "no" for blocked arteries
- Numeric data (weight): Use median value from same outcome group
- Median weight for no heart disease patients: 167.5
- Initial guess: 167.5 for weight
Proximity Matrix and Similarity Determination
- Step 1: Build random forest
- Step 2: Run all data through all trees
- Similarity defined: samples ending in same leaf node are similar
- Proximity matrix: rows and columns for each sample
- Increment by 1 when samples end in same leaf node
- Track across all trees in forest
Proximity Matrix Calculation Process
- Tree 1: Samples 3 and 4 end in same leaf → add 1 to matrix
- Tree 2: Samples 2, 3, and 4 end in same leaf → increment relevant pairs
- Tree 3: Only samples 3 and 4 end in same leaf → increment again
- Continue for all trees (example uses 10 trees)
- Final step: Divide each proximity value by total number of trees
Weighted Frequency Calculation for Categorical Data
- Calculate weighted frequency for "yes" and "no" using proximity values as weights
- For "yes":
- Frequency: 1/3 of samples
- Weight: proximity for "yes" samples ÷ sum of all proximities = 0.1
- Weighted frequency: 0.03
- For "no":
- Frequency: 2/3 of samples
- Weight: proximity for "no" samples ÷ sum of all proximities = 0.9
- Weighted frequency: 0.6
- Result: "no" selected due to higher weighted frequency
Weighted Average Calculation for Numeric Data
- Use proximities to calculate weighted average for weight
- Sample 1 weighted contribution: proximity/sum × weight
- Sample 2 (180 lbs) and Sample 3 (210 lbs) weighted contributions calculated
- Final weighted average weight: 198.5
Iterative Refinement Process
- Repeat entire process 6-7 times until convergence
- Each iteration: rebuild random forest → recalculate proximities → recalculate missing values
- Stop when missing values no longer change
Distance Matrix and Visualization Applications
- Proximity matrix before division by tree count
- Perfect similarity (samples in same leaf all trees) = proximity of 1.0
- Distance = 1 - proximity value
- Applications:
- Heat maps for data visualization
- MDS (Multidimensional Scaling) plots
- Works with any data type (ranks, multiple-choice, numeric) if trees can be built
Type 2 Missing Data: New Sample Classification
- Scenario: Pre-built random forest, new patient needs classification
- New patient missing blocked arteries status
- Method:
- Create two copies: one with heart disease, one without
- Use iterative method to guess missing values for both copies
- Run both samples through forest trees
- Count correct classifications for each option
- Example results:
- Heart disease option: correctly labeled in 3/3 trees
- No heart disease option: correctly labeled in 1/3 trees
- Select option with more correct classifications
Conclusion and Resources
- Summary of missing data handling and sample classification
- References to additional StatQuest content for heat maps and MDS plots
- Support options: subscriptions, Patreon, merchandise, donations
