AI summary by TwinMind

Live 2020-02-17!!! Imbalanced Data and Post-Hoc Tests

StatQuest with Josh Starmer

30m 53s

Introduction and Topics Overview

  • StatQuest livestream covering two main topics
  • Primary focus: imbalanced data handling
  • Secondary focus: post-hoc tests for ANOVA
  • Host taking deep breaths to calm down before starting

Imbalanced Data Problem Definition

  • Question from Alexander Romanchenko about unbalanced data
  • Example scenario: predicting if people love movie Troll 2
  • 90% of training data loves Troll 2
  • Problem: simply classifying everyone as "loves Troll 2" = 90% accuracy
  • More serious example: rare but contagious disease prediction
    • Using age and exposure as predictors
    • Predicting "no disease" for everyone = high accuracy but misses all disease cases
    • Consequences: infected people spread disease undetected

Machine Learning Methods and Expectations

  • Most ML methods expect relatively equal proportion of yes/no cases
  • Logistic regression, XGBoost affected by imbalance
  • Importance of examining confusion matrix output
  • Accuracy can be misleading with imbalanced data

Confusion Matrix Analysis

  • Example: 90% accuracy but 0% precision
  • Precision = true positives / (true positives + false positives)
  • Zero precision indicates problem with data balance
  • Confusion matrix reveals method weaknesses

Solutions for Imbalanced Data

Data Modification Approaches

  • Under-sampling/Down-scaling:
    • Randomly remove over-represented observations
    • Remove people without disease to emphasize disease classification
    • Results in closer to 50/50 split
  • Over-sampling/Up-scaling:
    • Duplicate under-represented individuals
    • Add copies of people with disease
    • Increases penalty for misclassification
  • Combined approach: Use both under and over-sampling

Advanced Sampling Methods

  • ROSE and SMOTE methods mentioned
  • More intelligent sampling vs random selection
  • Left as exercise for reader
  • Potential future livestream topic

Cross-Validation Considerations

  • Apply cross-validation BEFORE sampling methods
  • Avoid data leakage
  • Split data first, then apply sampling to each fold
  • Previous livestream covered data linkage in detail
  • Reference: StatQuest website video index

Weighting Observations

  • Assign larger weights to rare cases
  • Example in Random Forest:
    • Weights affect split decisions
    • Weights affect voting in classification
    • Individual votes count more for rare disease cases

Method Selection

  • Try different ML methods after balancing
  • Different methods may work better with imbalanced data
  • XGBoost worked best in example
  • Use confusion matrix to evaluate each method

Random Forest Built-in Balancing

  • Creates bootstrap datasets for each tree
  • Can be configured to balance classes per tree
  • Takes all minority class + equal number from majority class
  • Build more trees to compensate for smaller training sets
  • Track performance improvement with number of trees
  • Stop when performance plateaus

ANOVA Post-Hoc Testing Background

  • Question from Christina Yu about post-hoc tests after significant ANOVA
  • Example: four diet groups measuring weight gain after 4 weeks
  • ANOVA tests if any diets made difference overall
  • Importance of planning statistical tests before seeing data
  • Avoid p-hacking by testing predetermined hypotheses

Post-Hoc Testing Requirements

  • ANOVA shows at least one mean is different
  • Doesn't identify which specific means differ
  • Need pairwise comparisons for all diet combinations
  • Six total comparisons needed: A-B, A-C, A-D, B-C, B-D, C-D
  • Called "all pairwise tests"

Multiple Testing Problem

  • More statistical tests = increased false positive risk
  • Standard practice: Tukey-Kramer test (since 1950s)
  • Modern alternative: False Discovery Rate (FDR)
  • FDR developed in late 1990s/early 2000s
  • FDR is "game-changer" compared to older methods like Bonferroni

Recommended Approaches

  • Traditional: Tukey-Kramer test
  • Modern: Pairwise t-tests with FDR correction (Benjamini-Hochberg method)
  • Choice often depends on statistician's training era
  • FDR more general, applicable to various multiple testing scenarios
  • Can use t-tests after ANOVA if p-values adjusted with FDR

Q&A Session Highlights

  • Sampling percentages: Use cross-validation to find optimal ratios, aim for closer to 1:1 when possible
  • Precision vs sensitivity: Host admits difficulty remembering terminology differences, recommends focusing on confusion matrix analysis
  • Job interview advice: Memorize precision/sensitivity/specificity definitions 30 minutes before interviews
  • Humorous anecdote about software company interview and sorting algorithms

Future Topics

  • P-hacking discussion planned for next livestream (first Monday in March)
  • Random Forest Python version in development (March/April release)
  • Potential ROSE/SMOTE methods coverage if requested

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome