AI summary by TwinMind
Neural Networks Part 7: Cross Entropy Derivatives and Backpropagation
StatQuest with Josh Starmer
22m 7s
Introduction and Prerequisites
- StatQuest neural networks part 7: cross-entropy derivatives and backpropagation
- Assumes understanding of:
- Main ideas behind neural networks and backpropagation
- Neural networks with multiple inputs/outputs
- Softmax function
- Cross-entropy concepts
Neural Network Architecture Overview
- Example network: iris species classification
- Inputs: petal and sepal width measurements (0-1 values)
- Architecture:
- Input layer → hidden layers with activation functions
- Blue bent surface scaled by weight -0.1
- Orange bent surface scaled by weight 1.5
- Surfaces combined to create crinkled surfaces for each output
- Output layer: softmax for probability predictions
- Three outputs: setosa, versicolor, virginica probabilities
Raw Output Generation Process
- Setosa output: blue surface + orange surface + bias b₃ = green crinkled surface
- Versicolor output: blue surface + orange surface + bias = red crinkled surface
- Virginica output: blue surface + orange surface + bias = purple crinkled surface
- Raw outputs fed through softmax function for final probabilities
Cross-Entropy Calculation Example
- Training data example with petal/sepal measurements for setosa
- Neural network produces raw outputs
- Softmax converts to probabilities: 0.57 for setosa
- Cross-entropy evaluation: 0.56
Derivative Analysis Framework
- Focus on bias b₃ optimization (affects setosa raw output)
- b₃ changes affect:
- Setosa softmax value (numerator and denominator)
- Versicolor and virginica softmax values (denominator only)
- Different observed species → different cross-entropy equations → different derivatives
Chain Rule Application
- Derivative of cross-entropy w.r.t. b₃ using chain rule:
- ∂(cross-entropy)/∂b₃ = ∂(cross-entropy)/∂(predicted_prob) × ∂(predicted_prob)/∂(raw_output) × ∂(raw_output)/∂b₃
Derivative for Setosa Case
- When observed species = setosa:
- ∂(cross-entropy)/∂(predicted_prob_setosa) = -1/predicted_prob_setosa
- ∂(predicted_prob_setosa)/∂(raw_output_setosa) = predicted_prob_setosa × (1 - predicted_prob_setosa)
- ∂(raw_output_setosa)/∂b₃ = 1
- Final simplified result: predicted_prob_setosa - 1
Derivative for Virginica Case
- When observed species = virginica:
- Uses predicted probability for virginica in cross-entropy
- Chain rule involves derivative w.r.t. raw output for setosa
- ∂(predicted_prob_virginica)/∂(raw_output_setosa) = -predicted_prob_setosa × predicted_prob_virginica
- Final simplified result: predicted_prob_setosa
General Derivative Patterns
- For versicolor case: same as virginica (predicted_prob_setosa)
- Pattern depends on which parameter being optimized:
- b₃ (setosa): derivatives in terms of predicted_prob_setosa
- b₄ (versicolor): would be in terms of predicted_prob_versicolor
- b₅ (virginica): would be in terms of predicted_prob_virginica
Backpropagation Implementation
- Training data: 3 observations
- Initial b₃ = -2
- Total cross-entropy = 2.67
- Optimization process:
- Calculate derivative for each observation
- Sum derivatives: -0.77
- Apply gradient descent with learning rate = 1
- Step size = -0.77
- New b₃ = -2 - (-0.77) = -1.23
- Repeat until convergence
- Final optimized b₃ = -0.03
Specific Calculations
- Observation 1 (setosa): predicted_prob_setosa = 0.15, derivative = 0.15 - 1 = -0.85
- Observation 2 (virginica): predicted_prob_setosa = 0.04, derivative = 0.04
- Observation 3 (versicolor): predicted_prob_setosa = 0.04, derivative = 0.04
- Total derivative = -0.85 + 0.04 + 0.04 = -0.77
