AI summary by TwinMind

The SoftMax Derivative, Step-by-Step!!!

StatQuest with Josh Starmer

7m 12s

Introduction and Prerequisites

  • Josh Starmer presentation on softmax derivative calculation
  • Assumes prior understanding of softmax fundamentals
  • References previous StatQuest video on softmax basics
  • Context: neural network predicting iris species (setosa, versicolor, virginica)

Softmax Function Review

  • Neural network output values "all over the place"
  • Softmax layer normalizes outputs to predicted probabilities
  • Focus on derivative of softmax for setosa with respect to raw output value

Derivative Calculation for Same Class (Setosa)

  • Softmax fraction equals predicted probability for setosa
  • Derivative requires quotient rule application
  • Quotient rule breakdown:
    • Derivative of numerator: e^(raw_setosa) with respect to raw_setosa = e^(raw_setosa)
    • Derivative of denominator: e^(raw_setosa) with respect to raw_setosa
    • Apply quotient rule formula

Algebraic Simplification (Same Class)

  • Split numerator into two terms after applying quotient rule
  • First term = predicted probability for setosa
  • Second term simplifies to 1 (same numerator/denominator)
  • Final formula: P(setosa) × (1 - P(setosa))
  • Numerical example: P(setosa) = 0.69, derivative = 0.21

Cross-Class Derivative (Setosa vs Versicolor)

  • Raw output values for other classes affect softmax output
  • Need derivatives with respect to versicolor and virginica
  • Key difference: taking derivative with respect to different class

Quotient Rule for Cross-Class

  • Derivative of numerator (e^raw_setosa) with respect to versicolor = 0
  • Versicolor not in numerator, so first term becomes zero
  • Only second term remains in quotient rule application

Cross-Class Algebraic Simplification

  • Split remaining numerator into separate terms
  • First term = negative predicted probability for setosa
  • Second term = predicted probability for versicolor
  • Final formula: -P(setosa) × P(versicolor)
  • Numerical examples:
    • P(setosa) = 0.69, P(versicolor) = 0.1, derivative = -0.07
    • P(setosa) = 0.69, P(virginica) = 0.21, derivative = -0.15

General Pattern Summary

  • Same class derivative: P(i) × (1 - P(i))
  • Cross-class derivative: -P(i) × P(j) where i ≠ j
  • Virginica calculation left as homework exercise

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome