AI summary by TwinMind
Machine Learning Fundamentals: Cross Validation
StatQuest with Josh Starmer
6m 4s
Introduction and Problem Setup
- Josh Stormer presenting StatQuest on cross-validation
- Example: Using variables (chest pain, blood circulation, etc.) to predict heart disease
- Goal: Determine which machine learning method works best
- Options: Logistic regression, K-nearest neighbors, support vector machines, many others
The Two-Fold Challenge with Data
- Need data for two purposes:
- Training: Estimate parameters for ML methods (e.g., curve shape for logistic regression)
- Testing: Evaluate how well methods work on new data
- Terrible approach: Use all data for training
- No data left for testing
- Can't assess performance on unseen data
- Slightly better approach: 75% training, 25% testing split
- Problem: Uncertain if this specific split is optimal
- What about first 25% for testing or middle blocks?
Cross-Validation Solution
- Uses all data blocks for testing, one at a time
- Process:
- Use first 3 blocks for training, last block for testing
- Track performance on test data
- Rotate: different combination of blocks for training, remaining block for testing
- Continue until every block used for testing once
- Compare methods based on overall performance across all test sets
- Example result: Support vector machine performed best, so use it
Types of Cross-Validation
- 4-fold cross-validation: Data divided into 4 blocks (example used)
- Leave-one-out cross-validation: Each individual sample/patient is a block (extreme case)
- 10-fold cross-validation: Most common in practice - divide data into 10 blocks
Tuning Parameters Application
- Cross-validation helps find optimal tuning parameters
- Example: Ridge regression tuning parameter
- Use 10-fold cross-validation to determine best parameter value
- Parameters not estimated but "guessed" and optimized through cross-validation
