AI summary by TwinMind
Live 2020-03-02!!! Virus Models and p-hacking
StatQuest with Josh Starmer
30m 40s
Opening and Acknowledgments
- Josh Dharma hosts StatQuest livestream wearing StatQuest t-shirt
- Thanks viewers who used super chat during previous livestream
- Acknowledges missing super chat shout-outs during last stream due to screen management
Question 1: Corona Virus Epidemic Modeling (Chi Yang)
- Asked about best models for predicting coronavirus epidemic growth
- Most commonly used models: differential equation-based models
- Factors considered:
- Social networks (real-world, not digital)
- Disease transmission characteristics (contagiousness, spread patterns)
- Differential equations described as:
- Pretty elaborate equations
- Fun and intuitive
- Computers do most computational work
- Josh's experience: took differential equation classes in grad school, found them enjoyable
- Current focus: statistics and machine learning (differential equations less popular in these areas)
- Upcoming content priorities: neural networks, time series, other topics
- May cover differential equations if heavily requested
- Referenced academic article on mathematical models for epidemics (technical but readable introduction/abstract)
Question 2: P-Hacking Explanation (Josh Star Murr)
- Context: previous livestream discussed ANOVA with four diet groups
- Measured weight gain after 4 weeks on different diets
- Diet C showed obvious difference from Diet A
- Problem: testing only groups that "look different" = P-hacking
P-Hacking Demonstration
- Simulated recovery time from illness using normal distribution
- 95% of area under curve between 5-15 days (2.5% each tail)
- Took multiple groups of 3 people from same distribution
- Tested pairs repeatedly:
- Most p-values > 0.05 (not significant)
- Eventually found p-value = 0.02 (false positive)
- Cherry-picking tests that look different leads to false positives
- No p-value adjustment works on single test
P-Hacking Definition
- Misuse and abuse of analysis techniques
- Results in being fooled by false positives
- Not necessarily malicious - legitimate temptation to test obvious differences
- Solution: plan tests before seeing data
Subtle Form of P-Hacking
- Example: p-value = 0.06 (just above 0.05 threshold)
- Temptation: add more data to get significant result
- Added one measurement per group → p-value = 0.02
- Still false positive (all data from same distribution)
- Josh's simulation results: borderline p-values (0.05-0.1)
- Added one measurement per group
- 1/3 became significant (33% false positive rate)
Power Analysis Solution
- Must do power analysis before first experiment
- Power = probability of correctly rejecting null hypothesis
- High separation between distributions = high power
- Low separation = low power
- Can increase power by increasing sample size
- 80% power analysis: determines sample size needed to correctly reject null hypothesis 80% of time
- When p-value just above 0.05: use current means/standard deviations to do proper power analysis rather than adding data
Q&A Session
Bayesian Statistics Question
- Josh wants to do Bayesian statistics content
- Concepts straightforward, implementation gets messy quickly
- Gets stuck in technical nuances during research
- Acknowledges many people use Bayesian methods successfully
- Promises eventual coverage, likely 2021 project
- Describes it as "mountain to climb and conquer"
Machine Learning Interview Tips
- Josh lacks direct experience with ML technical interviews
- Proposes community contribution: post actual interview questions
- Suggests livestream format answering interview questions
- Acknowledges his answers may not match what interviewers want
- References previous livestream example of interview that went poorly due to disagreeing with question
Random vs Fixed Effects
- High priority on to-do list
- Described as "itch for years"
Future Vision for StatQuest
- Long-term dream: expand like Khan Academy
- Currently solo operation limiting output
- Fantasy: team of "StatQuesters" covering more topics
- 10-year vision for comprehensive coverage
Teaching Philosophy and Tips
- Statistics/ML actually hard for Josh personally
- Teaching method: explain to himself step-by-step
- Can't skip big steps in explanation
- Previous experience: monthly seminars for biologists
- Biologists often not math-oriented, sat far back looking stunned
- Used facial expressions to gauge understanding
- Made mental notes when concepts weren't clear
