AI summary by TwinMind
Support Vector Machines Part 2: The Polynomial Kernel (Part 2 of 3)
StatQuest with Josh Starmer
7m 15s
Introduction and Context
- Educational video by Josh Stormer on StatQuest
- Focus: Support vector machines part 2, specifically polynomial kernel
- Prerequisites: Assumes familiarity with basic support vector machines
- Reference to previous StatQuest video (link in description)
Training Data Example
- Dataset: Drug dosage measurements from patients
- Red dots: Patients not cured
- Green dots: Patients cured
- Problem: Drug only works with correct dosage (not too small/large)
- Challenge: Training data had significant overlap, couldn't find satisfying linear support vector classifier
Polynomial Kernel Solution
- Added Y-axis coordinate by squaring original dosage measurements
- Enabled linear separation in higher dimension
- Used support vector machine with polynomial kernel
- Successfully computed relationships between observations in higher dimension
- Found good support vector classifier based on high-dimensional relationships
Polynomial Kernel Formula
- Formula: (a·b + r)^d
- Parameters:
- a, b: Two different observations in dataset
- r: Coefficient of polynomial
- d: Degree of polynomial
First Example (r=1/2, d=2)
- Set r = 1/2, d = 2
- Expanded polynomial through multiplication
- Resulted in dot product with three terms
- High-dimensional coordinates:
- X-axis: Original coordinates
- Y-axis: Squared original coordinates
- Z-axis: Constant (can be ignored)
Second Example (r=1, d=2)
- Set r = 1, d = 2
- Generated different polynomial and dot product
- New coordinates:
- X-axis: √2 × original dosage values
- Y-axis: Original dosage values squared
- Z-axis: Constant (ignored)
- Both examples enable support vector classifier in higher dimension
Key Insight - Kernel Trick
- Only need to calculate dot products between point pairs
- Don't need to actually transform data to higher dimensions
- Kernel directly computes high-dimensional relationships
- Example calculation: Plugging dosages into kernel yields 2.25 as relationship value
- Provides all necessary information for support vector classifier
Parameter Selection
- r and d values determined using cross-validation
- Once parameters selected, plug observations into formula
- Calculate high-dimensional relationships through kernel math
Conclusion
- Polynomial kernel enables SVM classification in cases where linear separation impossible
- Kernel trick avoids explicit high-dimensional transformation
- Cross-validation determines optimal parameters
