AI summary by TwinMind
Support Vector Machines Part 3: The Radial (RBF) Kernel (Part 3 of 3)
StatQuest with Josh Starmer
15m 52s
Introduction & Context
- StatQuest video by Josh Starmer on Support Vector Machines Part 3: Radial Kernel
- Assumes familiarity with SVMs and polynomial kernel
- Focus on radial kernel parameters, high-dimensional relationships, infinite dimensions
Training Data Problem
- Drug dosage dataset: red dots (not cured), green dots (cured)
- Drug only works with correct dosage (not too small/large)
- Overlapping data prevented satisfying support vector classifier
- Solution: Support Vector Machine with radial kernel (RBF - Radial Basis Function)
Radial Kernel Behavior
- Finds support vector classifiers in infinite dimensions
- Behaves like weighted nearest neighbor model
- Closest observations (nearest neighbors) = high influence on classification
- Distant observations = little influence on classification
- Uses closest observations' classifications for new observation classification
Radial Kernel Mathematics
- Variables a and b = two different dosage measurements
- Squared distance between observations calculated
- Influence = function of squared distance
- Gamma parameter (determined by cross-validation) scales squared distance
- Example calculations:
- Gamma = 1, close observations: result = 0.11
- Gamma = 2, same observations: result = 0.01 (less influence)
- Gamma = 1, distant observations: result ≈ 0 (minimal influence)
- Output represents high-dimensional relationship between observations
Polynomial Kernel Intuition
- Simplified training dataset to 2 observations
- R = 0 polynomial kernel analysis:
- D = 2: single coordinate = original measurement squared
- D = 3: single coordinate = original measurement cubed
- D = 1: data stays in original location
- Adding multiple polynomial kernels (R = 0):
- D = 1 + D = 2: 2-dimensional coordinates
- D = 1 + D = 2 + D = 3: 3-dimensional coordinates
- Continuing to D = infinity: infinite dimensional coordinates
Mathematical Derivation (Advanced)
- Radial kernel mathematical expansion
- Set gamma = 1/2 to simplify calculations
- Taylor series expansion introduction:
- Function f(X) = infinite sum
- e^X Taylor series derivation
- Setting a = 0, e^0 = 1
- Final form: e^X = 1 + X + X²/2! + X³/3! + ...
Radial Kernel as Infinite Sum
- Taylor series expansion of e^(aB) term
- Each term contains polynomial kernel with R = 0, D = 0 to infinity
- Polynomial kernel equivalents:
- R = 0, D = 0: equals 1
- R = 0, D = 1: equals a × b
- R = 0, D = 2: equals a × b²
- Dot product conversion for e^(aB)
- Final radial kernel = dot product with infinite dimensional coordinates
- S = square root of first term for complete dot product form
Conclusion
- Radial kernel equals dot product with infinite dimensional coordinates
- Plugging numbers into radial kernel yields high-dimensional relationship between observations
