AI summary by TwinMind
StatQuest: Random Forests Part 1 - Building, Using and Evaluating
StatQuest with Josh Starmer
9m 54s
Introduction to Random Forests
- Decision trees: easy to build, use, interpret but lack accuracy
- Quote from "Elements of Statistical Learning" (machine learning Bible): trees have accuracy issues preventing them from being ideal predictive tools
- Trees work great with training data but inflexible with new samples
- Random forests combine decision tree simplicity with flexibility for vast accuracy improvement
Step 1: Creating Bootstrap Dataset
- Example uses 4-sample dataset (artificially small)
- Bootstrap dataset creation: randomly select samples from original with replacement
- Same sample can be picked multiple times
- Demonstration: 4 random selections created bootstrap dataset with duplicates
Step 2: Building Decision Tree with Variable Subset
- Create decision tree using bootstrap dataset
- Only use random subset of variables/columns at each step
- Example: consider only 2 variables at each step (instead of all 4)
- Root node selection: randomly chose "good blood circulation" and "blocked arteries" as candidates
- "Good blood circulation" performed best for sample separation
- Continue tree building considering random subset at each node split
- Process creates tree using: 1) bootstrap dataset, 2) random variable subsets
Random Forest Construction Process
- Repeat steps 1-2 multiple times (ideally hundreds)
- Example shows 6 trees for space constraints
- Bootstrap sampling + variable subset selection creates wide tree variety
- Variety makes random forests more effective than individual decision trees
Using Random Forest for Prediction
- New patient example: determine heart disease presence
- Run patient data through each tree in forest
- Track each tree's prediction (yes/no votes)
- Final prediction: majority vote across all trees
- Example: "yes" received most votes → conclude patient has heart disease
- Terminology: bootstrapping + aggregation = "bagging"
Evaluating Random Forest Accuracy
- Bootstrap sampling creates duplicates, leaving ~1/3 original data unused
- Unused data = "out-of-bag" dataset (not "out-of-boot" as presenter would prefer)
- Out-of-bag samples not used in tree creation
- Run out-of-bag samples through trees built without them
- Example results: some trees correctly/incorrectly labeled out-of-bag samples
- Final out-of-bag prediction: majority vote
- Accuracy measurement: proportion of correctly classified out-of-bag samples
- Out-of-bag error: proportion incorrectly classified
Optimization Process
- Compare different variable counts per step (e.g., 2 vs 3 variables)
- Test multiple settings, choose most accurate random forest
- Process: 1) build random forest, 2) estimate accuracy, 3) adjust parameters, 4) repeat
- Starting point: typically square root of total variables
- Try settings above/below starting value
