AI summary by TwinMind

Machine Learning Fundamentals: Cross Validation

StatQuest with Josh Starmer

6m 4s

Introduction and Problem Setup

  • Josh Stormer presenting StatQuest on cross-validation
  • Example: Using variables (chest pain, blood circulation, etc.) to predict heart disease
  • Goal: Determine which machine learning method works best
  • Options: Logistic regression, K-nearest neighbors, support vector machines, many others

The Two-Fold Challenge with Data

  • Need data for two purposes:
    • Training: Estimate parameters for ML methods (e.g., curve shape for logistic regression)
    • Testing: Evaluate how well methods work on new data
  • Terrible approach: Use all data for training
    • No data left for testing
    • Can't assess performance on unseen data
  • Slightly better approach: 75% training, 25% testing split
    • Problem: Uncertain if this specific split is optimal
    • What about first 25% for testing or middle blocks?

Cross-Validation Solution

  • Uses all data blocks for testing, one at a time
  • Process:
    • Use first 3 blocks for training, last block for testing
    • Track performance on test data
    • Rotate: different combination of blocks for training, remaining block for testing
    • Continue until every block used for testing once
  • Compare methods based on overall performance across all test sets
  • Example result: Support vector machine performed best, so use it

Types of Cross-Validation

  • 4-fold cross-validation: Data divided into 4 blocks (example used)
  • Leave-one-out cross-validation: Each individual sample/patient is a block (extreme case)
  • 10-fold cross-validation: Most common in practice - divide data into 10 blocks

Tuning Parameters Application

  • Cross-validation helps find optimal tuning parameters
  • Example: Ridge regression tuning parameter
  • Use 10-fold cross-validation to determine best parameter value
  • Parameters not estimated but "guessed" and optimized through cross-validation

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome