Live 2020-01-20!!! Favorite ML, Data Leakage, How to Learn ML

StatQuest with Josh Starmer

Introduction & Stream Format

Question: What is your favorite machine learning algorithm?
Answer: Random Forests
Reasoning for Random Forests:
- Provides insight into underlying mechanisms/structure vs pure accuracy
- Easy to interpret - shows what variables doing and why
- Handles all data types: continuous, discrete, categorical
- Provides clustering and visualization of sample relationships
- Simple confidence estimation method
Confidence Calculation Example:
- 100 trees example: if all agree → high confidence
- If only 51% agree → lower confidence
- Easy to count tree agreement for decision confidence
Acknowledges other algorithms are interesting but Random Forests best for his research work

Definition: When training and testing datasets are not independent
Obvious Example: Using same data for training and testing
- Causes overestimation of model performance
Proper Approach:
- Split original data into independent training/testing subsets
- Train on training data, test on testing data
Sneaky Data Leakage Example: Missing data imputation
- Wrong: Impute missing values first, then split data
- Problem: Testing data involved in filling training data values
- Correct: Split data first, then impute missing values separately for each subset
- Ensures training/testing independence

Question: Where do you learn nitty-gritty details?
Answer: Internet/Google research
Process:
- Read everything available on topic
- Often don't understand initially - complicated math, notation, unfamiliar terms
- Keep reading multiple sources
- Google unfamiliar words/terms
- Gradually build understanding
- Return to original sources - make more sense second time
- Iterative process of reading, googling, re-reading
Timeline Examples:
- Can take days to months to years
- PCA video: ~2 years of research
- Not full-time work but consistent daily reading
Verification Process:
- Annotate materials
- Redo math personally
- Derive everything to ensure understanding
- Run basic datasets through algorithms
- Verify step-by-step matches paper calculations
Current Work: XG Boost videos - 2+ more coming with math details and optimizations

R-squared Discussion:
- Never seen R-squared > 1 personally
- R-squared < 0 possible when model worse than simple model
- Adjusted R-squared: Compensates for adding variables
- Problem: Adding random variables can increase R-squared by chance
- Adjusted R-squared penalizes for number of variables used
Video Difficulty Classification:
- Website (statquest.org) has video index
- Already ordered simple to complicated within categories
- Basics section starts with most basic concepts
- Machine learning section progresses from simple to complex
Learning Formulas vs Software:
- Depends on goals: user vs power user vs algorithm developer
- For practitioners: main ideas most important (executive summary approach)
- Math helpful for understanding strengths/weaknesses
- Full derivation knowledge needed for developing new algorithms
Book Recommendation: Introduction to Statistical Learning
- Contains calculus but can skip detailed math
- Read introductions and examples without calculus
- Covers 90% of needed knowledge
- Available free download

Next Month: XG Boost completion (2+ videos on math details, optimizations)
Housekeeping Videos:
- Updated p-value video (2-part series)
- Possibly naive Bayes
- Pearson vs Spearman correlation
Major Topics Planned:
- Neural networks (after current work)
- Time series (summer/early fall)
Research Phase: Will spend time researching neural networks for unique StatQuest approach

Try for free