AI summary by TwinMind
The SoftMax Derivative, Step-by-Step!!!
StatQuest with Josh Starmer
7m 12s
Introduction and Prerequisites
- Josh Starmer presentation on softmax derivative calculation
- Assumes prior understanding of softmax fundamentals
- References previous StatQuest video on softmax basics
- Context: neural network predicting iris species (setosa, versicolor, virginica)
Softmax Function Review
- Neural network output values "all over the place"
- Softmax layer normalizes outputs to predicted probabilities
- Focus on derivative of softmax for setosa with respect to raw output value
Derivative Calculation for Same Class (Setosa)
- Softmax fraction equals predicted probability for setosa
- Derivative requires quotient rule application
- Quotient rule breakdown:
- Derivative of numerator: e^(raw_setosa) with respect to raw_setosa = e^(raw_setosa)
- Derivative of denominator: e^(raw_setosa) with respect to raw_setosa
- Apply quotient rule formula
Algebraic Simplification (Same Class)
- Split numerator into two terms after applying quotient rule
- First term = predicted probability for setosa
- Second term simplifies to 1 (same numerator/denominator)
- Final formula: P(setosa) × (1 - P(setosa))
- Numerical example: P(setosa) = 0.69, derivative = 0.21
Cross-Class Derivative (Setosa vs Versicolor)
- Raw output values for other classes affect softmax output
- Need derivatives with respect to versicolor and virginica
- Key difference: taking derivative with respect to different class
Quotient Rule for Cross-Class
- Derivative of numerator (e^raw_setosa) with respect to versicolor = 0
- Versicolor not in numerator, so first term becomes zero
- Only second term remains in quotient rule application
Cross-Class Algebraic Simplification
- Split remaining numerator into separate terms
- First term = negative predicted probability for setosa
- Second term = predicted probability for versicolor
- Final formula: -P(setosa) × P(versicolor)
- Numerical examples:
- P(setosa) = 0.69, P(versicolor) = 0.1, derivative = -0.07
- P(setosa) = 0.69, P(virginica) = 0.21, derivative = -0.15
General Pattern Summary
- Same class derivative: P(i) × (1 - P(i))
- Cross-class derivative: -P(i) × P(j) where i ≠ j
- Virginica calculation left as homework exercise
