AI summary by TwinMind
Live 2020-02-17!!! Imbalanced Data and Post-Hoc Tests
StatQuest with Josh Starmer
30m 53s
Introduction and Topics Overview
- StatQuest livestream covering two main topics
- Primary focus: imbalanced data handling
- Secondary focus: post-hoc tests for ANOVA
- Host taking deep breaths to calm down before starting
Imbalanced Data Problem Definition
- Question from Alexander Romanchenko about unbalanced data
- Example scenario: predicting if people love movie Troll 2
- 90% of training data loves Troll 2
- Problem: simply classifying everyone as "loves Troll 2" = 90% accuracy
- More serious example: rare but contagious disease prediction
- Using age and exposure as predictors
- Predicting "no disease" for everyone = high accuracy but misses all disease cases
- Consequences: infected people spread disease undetected
Machine Learning Methods and Expectations
- Most ML methods expect relatively equal proportion of yes/no cases
- Logistic regression, XGBoost affected by imbalance
- Importance of examining confusion matrix output
- Accuracy can be misleading with imbalanced data
Confusion Matrix Analysis
- Example: 90% accuracy but 0% precision
- Precision = true positives / (true positives + false positives)
- Zero precision indicates problem with data balance
- Confusion matrix reveals method weaknesses
Solutions for Imbalanced Data
Data Modification Approaches
- Under-sampling/Down-scaling:
- Randomly remove over-represented observations
- Remove people without disease to emphasize disease classification
- Results in closer to 50/50 split
- Over-sampling/Up-scaling:
- Duplicate under-represented individuals
- Add copies of people with disease
- Increases penalty for misclassification
- Combined approach: Use both under and over-sampling
Advanced Sampling Methods
- ROSE and SMOTE methods mentioned
- More intelligent sampling vs random selection
- Left as exercise for reader
- Potential future livestream topic
Cross-Validation Considerations
- Apply cross-validation BEFORE sampling methods
- Avoid data leakage
- Split data first, then apply sampling to each fold
- Previous livestream covered data linkage in detail
- Reference: StatQuest website video index
Weighting Observations
- Assign larger weights to rare cases
- Example in Random Forest:
- Weights affect split decisions
- Weights affect voting in classification
- Individual votes count more for rare disease cases
Method Selection
- Try different ML methods after balancing
- Different methods may work better with imbalanced data
- XGBoost worked best in example
- Use confusion matrix to evaluate each method
Random Forest Built-in Balancing
- Creates bootstrap datasets for each tree
- Can be configured to balance classes per tree
- Takes all minority class + equal number from majority class
- Build more trees to compensate for smaller training sets
- Track performance improvement with number of trees
- Stop when performance plateaus
ANOVA Post-Hoc Testing Background
- Question from Christina Yu about post-hoc tests after significant ANOVA
- Example: four diet groups measuring weight gain after 4 weeks
- ANOVA tests if any diets made difference overall
- Importance of planning statistical tests before seeing data
- Avoid p-hacking by testing predetermined hypotheses
Post-Hoc Testing Requirements
- ANOVA shows at least one mean is different
- Doesn't identify which specific means differ
- Need pairwise comparisons for all diet combinations
- Six total comparisons needed: A-B, A-C, A-D, B-C, B-D, C-D
- Called "all pairwise tests"
Multiple Testing Problem
- More statistical tests = increased false positive risk
- Standard practice: Tukey-Kramer test (since 1950s)
- Modern alternative: False Discovery Rate (FDR)
- FDR developed in late 1990s/early 2000s
- FDR is "game-changer" compared to older methods like Bonferroni
Recommended Approaches
- Traditional: Tukey-Kramer test
- Modern: Pairwise t-tests with FDR correction (Benjamini-Hochberg method)
- Choice often depends on statistician's training era
- FDR more general, applicable to various multiple testing scenarios
- Can use t-tests after ANOVA if p-values adjusted with FDR
Q&A Session Highlights
- Sampling percentages: Use cross-validation to find optimal ratios, aim for closer to 1:1 when possible
- Precision vs sensitivity: Host admits difficulty remembering terminology differences, recommends focusing on confusion matrix analysis
- Job interview advice: Memorize precision/sensitivity/specificity definitions 30 minutes before interviews
- Humorous anecdote about software company interview and sorting algorithms
Future Topics
- P-hacking discussion planned for next livestream (first Monday in March)
- Random Forest Python version in development (March/April release)
- Potential ROSE/SMOTE methods coverage if requested
