AI summary by TwinMind

Neural Networks Part 5: ArgMax and SoftMax

StatQuest with Josh Starmer

14m 3s

Introduction and Prerequisites

  • StatQuest Part 5 on neural networks covering ArgMax and SoftMax functions
  • Assumes understanding of:
    • Main neural network concepts
    • Back propagation fundamentals
    • Neural networks with multiple input/output nodes
  • Previous example: iris classification using petal/sepal widths predicting setosa, versicolor, virginica species

Raw Neural Network Output Issues

  • Example prediction: iris classified as versicolor (output value 0.86 closest to 1)
  • New example with petal width = 0, sepal width = 1:
    • Setosa: 1.43
    • Versicolor: -0.4
    • Virginica: 0.23
  • Problems with raw outputs:
    • Values not constrained between 0 and 1
    • Can exceed 1 (setosa = 1.43)
    • Can be negative (versicolor = -0.4)
    • Makes interpretation difficult

ArgMax Function

  • Sets largest value to 1, all others to 0
  • Example application:
    • Setosa (1.43) → 1
    • Versicolor (-0.4) → 0
    • Virginica (0.23) → 0
  • Advantages: extremely easy to interpret
  • Critical limitation: cannot optimize weights/biases
    • Outputs are constants (0 and 1)
    • Derivatives are zero (flat slopes)
    • Chain rule produces zero derivatives
    • Gradient descent cannot step toward optimal parameters
    • Unusable for back propagation

SoftMax Function Introduction

  • Alternative to ArgMax for training
  • Often used: SoftMax for training, ArgMax for final output
  • Represented by soft teddy bear symbol

SoftMax Calculations

  • Formula: e^(raw_output_i) / sum(e^(raw_output_j) for all j)
  • Setosa calculation:
    • e^1.43 / (e^1.43 + e^(-0.4) + e^0.23) = 0.69
  • Versicolor calculation:
    • e^(-0.4) / (e^1.43 + e^(-0.4) + e^0.23) = 0.1
  • Virginica calculation:
    • e^0.23 / (e^1.43 + e^(-0.4) + e^0.23) = 0.21

SoftMax Properties

  • Preserves ranking: largest raw → largest SoftMax (1.43→0.69), second largest raw → second largest SoftMax (0.23→0.21), smallest raw → smallest SoftMax (-0.4→0.1)
  • All outputs between 0 and 1 (guaranteed regardless of input count)
  • Sum equals 1 (0.69 + 0.1 + 0.21 = 1)
  • Can be interpreted as "predicted probabilities" (with caveats)

Predicted Probabilities Caveat

  • Probabilities depend on weights/biases
  • Weights/biases depend on random initial values
  • Different initial values → different weights/biases → different raw outputs → different SoftMax outputs
  • Same classification accuracy but different "probabilities"
  • Don't trust accuracy of predicted probabilities

SoftMax Derivatives for Back Propagation

  • General form: SoftMax_i = e^(raw_i) / sum(e^(raw_j))
  • Derivative of predicted probability for setosa w.r.t. setosa raw output:
    • P_setosa × (1 - P_setosa) = 0.69 × (1 - 0.69) = 0.21
  • Derivative w.r.t. versicolor raw output:
    • -P_setosa × P_versicolor = -0.69 × 0.10 = -0.07
  • Derivative w.r.t. virginica raw output:
    • -P_setosa × P_virginica = -0.69 × 0.21 = -0.15
  • Unlike ArgMax, derivatives are non-zero and usable for gradient descent

Training vs Prediction Strategy

  • Training: use SoftMax (enables back propagation)
  • Prediction: use ArgMax (easy interpretation)
  • SoftMax enables optimization, ArgMax provides clear classification

Loss Function Consideration

  • Previous StatQuest used sum of squared residuals
  • With SoftMax outputs (probabilities 0-1), typically use cross entropy loss
  • Cross entropy topic for next StatQuest episode

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome