AI summary by TwinMind

Neural Networks Part 7: Cross Entropy Derivatives and Backpropagation

StatQuest with Josh Starmer

22m 7s

Introduction and Prerequisites

  • StatQuest neural networks part 7: cross-entropy derivatives and backpropagation
  • Assumes understanding of:
    • Main ideas behind neural networks and backpropagation
    • Neural networks with multiple inputs/outputs
    • Softmax function
    • Cross-entropy concepts

Neural Network Architecture Overview

  • Example network: iris species classification
  • Inputs: petal and sepal width measurements (0-1 values)
  • Architecture:
    • Input layer → hidden layers with activation functions
    • Blue bent surface scaled by weight -0.1
    • Orange bent surface scaled by weight 1.5
    • Surfaces combined to create crinkled surfaces for each output
  • Output layer: softmax for probability predictions
  • Three outputs: setosa, versicolor, virginica probabilities

Raw Output Generation Process

  • Setosa output: blue surface + orange surface + bias b₃ = green crinkled surface
  • Versicolor output: blue surface + orange surface + bias = red crinkled surface
  • Virginica output: blue surface + orange surface + bias = purple crinkled surface
  • Raw outputs fed through softmax function for final probabilities

Cross-Entropy Calculation Example

  • Training data example with petal/sepal measurements for setosa
  • Neural network produces raw outputs
  • Softmax converts to probabilities: 0.57 for setosa
  • Cross-entropy evaluation: 0.56

Derivative Analysis Framework

  • Focus on bias b₃ optimization (affects setosa raw output)
  • b₃ changes affect:
    • Setosa softmax value (numerator and denominator)
    • Versicolor and virginica softmax values (denominator only)
  • Different observed species → different cross-entropy equations → different derivatives

Chain Rule Application

  • Derivative of cross-entropy w.r.t. b₃ using chain rule:
    • ∂(cross-entropy)/∂b₃ = ∂(cross-entropy)/∂(predicted_prob) × ∂(predicted_prob)/∂(raw_output) × ∂(raw_output)/∂b₃

Derivative for Setosa Case

  • When observed species = setosa:
    • ∂(cross-entropy)/∂(predicted_prob_setosa) = -1/predicted_prob_setosa
    • ∂(predicted_prob_setosa)/∂(raw_output_setosa) = predicted_prob_setosa × (1 - predicted_prob_setosa)
    • ∂(raw_output_setosa)/∂b₃ = 1
  • Final simplified result: predicted_prob_setosa - 1

Derivative for Virginica Case

  • When observed species = virginica:
    • Uses predicted probability for virginica in cross-entropy
    • Chain rule involves derivative w.r.t. raw output for setosa
    • ∂(predicted_prob_virginica)/∂(raw_output_setosa) = -predicted_prob_setosa × predicted_prob_virginica
  • Final simplified result: predicted_prob_setosa

General Derivative Patterns

  • For versicolor case: same as virginica (predicted_prob_setosa)
  • Pattern depends on which parameter being optimized:
    • b₃ (setosa): derivatives in terms of predicted_prob_setosa
    • b₄ (versicolor): would be in terms of predicted_prob_versicolor
    • b₅ (virginica): would be in terms of predicted_prob_virginica

Backpropagation Implementation

  • Training data: 3 observations
  • Initial b₃ = -2
  • Total cross-entropy = 2.67
  • Optimization process:
    • Calculate derivative for each observation
    • Sum derivatives: -0.77
    • Apply gradient descent with learning rate = 1
    • Step size = -0.77
    • New b₃ = -2 - (-0.77) = -1.23
    • Repeat until convergence
  • Final optimized b₃ = -0.03

Specific Calculations

  • Observation 1 (setosa): predicted_prob_setosa = 0.15, derivative = 0.15 - 1 = -0.85
  • Observation 2 (virginica): predicted_prob_setosa = 0.04, derivative = 0.04
  • Observation 3 (versicolor): predicted_prob_setosa = 0.04, derivative = 0.04
  • Total derivative = -0.85 + 0.04 + 0.04 = -0.77

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome