AI summary by TwinMind
Neural Networks Part 5: ArgMax and SoftMax
StatQuest with Josh Starmer
14m 3s
Introduction and Prerequisites
- StatQuest Part 5 on neural networks covering ArgMax and SoftMax functions
- Assumes understanding of:
- Main neural network concepts
- Back propagation fundamentals
- Neural networks with multiple input/output nodes
- Previous example: iris classification using petal/sepal widths predicting setosa, versicolor, virginica species
Raw Neural Network Output Issues
- Example prediction: iris classified as versicolor (output value 0.86 closest to 1)
- New example with petal width = 0, sepal width = 1:
- Setosa: 1.43
- Versicolor: -0.4
- Virginica: 0.23
- Problems with raw outputs:
- Values not constrained between 0 and 1
- Can exceed 1 (setosa = 1.43)
- Can be negative (versicolor = -0.4)
- Makes interpretation difficult
ArgMax Function
- Sets largest value to 1, all others to 0
- Example application:
- Setosa (1.43) → 1
- Versicolor (-0.4) → 0
- Virginica (0.23) → 0
- Advantages: extremely easy to interpret
- Critical limitation: cannot optimize weights/biases
- Outputs are constants (0 and 1)
- Derivatives are zero (flat slopes)
- Chain rule produces zero derivatives
- Gradient descent cannot step toward optimal parameters
- Unusable for back propagation
SoftMax Function Introduction
- Alternative to ArgMax for training
- Often used: SoftMax for training, ArgMax for final output
- Represented by soft teddy bear symbol
SoftMax Calculations
- Formula: e^(raw_output_i) / sum(e^(raw_output_j) for all j)
- Setosa calculation:
- e^1.43 / (e^1.43 + e^(-0.4) + e^0.23) = 0.69
- Versicolor calculation:
- e^(-0.4) / (e^1.43 + e^(-0.4) + e^0.23) = 0.1
- Virginica calculation:
- e^0.23 / (e^1.43 + e^(-0.4) + e^0.23) = 0.21
SoftMax Properties
- Preserves ranking: largest raw → largest SoftMax (1.43→0.69), second largest raw → second largest SoftMax (0.23→0.21), smallest raw → smallest SoftMax (-0.4→0.1)
- All outputs between 0 and 1 (guaranteed regardless of input count)
- Sum equals 1 (0.69 + 0.1 + 0.21 = 1)
- Can be interpreted as "predicted probabilities" (with caveats)
Predicted Probabilities Caveat
- Probabilities depend on weights/biases
- Weights/biases depend on random initial values
- Different initial values → different weights/biases → different raw outputs → different SoftMax outputs
- Same classification accuracy but different "probabilities"
- Don't trust accuracy of predicted probabilities
SoftMax Derivatives for Back Propagation
- General form: SoftMax_i = e^(raw_i) / sum(e^(raw_j))
- Derivative of predicted probability for setosa w.r.t. setosa raw output:
- P_setosa × (1 - P_setosa) = 0.69 × (1 - 0.69) = 0.21
- Derivative w.r.t. versicolor raw output:
- -P_setosa × P_versicolor = -0.69 × 0.10 = -0.07
- Derivative w.r.t. virginica raw output:
- -P_setosa × P_virginica = -0.69 × 0.21 = -0.15
- Unlike ArgMax, derivatives are non-zero and usable for gradient descent
Training vs Prediction Strategy
- Training: use SoftMax (enables back propagation)
- Prediction: use ArgMax (easy interpretation)
- SoftMax enables optimization, ArgMax provides clear classification
Loss Function Consideration
- Previous StatQuest used sum of squared residuals
- With SoftMax outputs (probabilities 0-1), typically use cross entropy loss
- Cross entropy topic for next StatQuest episode
