AI summary by TwinMind
Lecture 7 Part 2: Second Derivatives, Bilinear Forms, and Hessian Matrices
MIT OpenCourseWare
46m 9s
Introduction to Second Derivatives
- Second derivatives = derivative of the derivative
- For functions from matrices to matrices, need careful consideration of what second derivative represents
- First derivative F'(x) takes small change dx, outputs df (first-order approximation)
- Second derivative naturally defined as derivative of F'
Linear Operators and Vector Spaces
- F'(x + dx') - F'(x) represents difference of two linear operators
- Linear operators form vector space
- Addition/subtraction of operators: (L1 + L2)(v) = L1(v) + L2(v)
- Scalar multiplication: (αL1)(v) = α(L1(v))
- Matrix operations derive from these linear operator properties
Bilinear Forms
- Second derivative F''(x) takes two inputs: dx' and dx
- F''(x)(dx', dx) = F'(x + dx') - F'(x)
- Bilinear form: linear in both arguments
- Properties: B(u1 + u2, v) = B(u1, v) + B(u2, v) and B(u, v1 + v2) = B(u, v1) + B(u, v2)
- Scaling both arguments: B(2u, 3v) = 6B(u, v)
Symmetry of Second Derivatives
- F''(x)(dx', dx) = F''(x)(dx, dx')
- Proof by expanding definition using commutativity of vector addition
- Writing out F'(x + dx')(dx) - F'(x)(dx) shows both dx and dx' appear symmetrically
- This generalizes the familiar property from single-variable calculus
Multivariable Calculus Example
- Scalar function f: ℝⁿ → ℝ
- F'(x) is row vector (transpose of gradient)
- F''(x) takes two vectors, outputs scalar
- Only way to do this: dx'ᵀ H dx where H is n×n matrix
- Must be symmetric matrix due to symmetry property
- H called Hessian matrix
Hessian Matrix Components
- Gradient ∇f = [∂f/∂x₁, ..., ∂f/∂xₙ]ᵀ
- D(∇f) = H dx where H is Jacobian of gradient
- Hij = ∂²f/∂xi∂xj
- Symmetry: Hij = Hji (can swap order of partial derivatives)
- This familiar 1802 property comes from general derivative definition
Matrix Determinant Example
- f(A) = det(A) where A is matrix
- F'(A) = det(A) · trace(A⁻¹ dA)
- Second derivative calculation using product rule and chain rule
- F''(A)(dA', dA) = det(A) · trace(A⁻¹ dA') · trace(A⁻¹ dA) - det(A) · trace(A⁻¹ dA' A⁻¹ dA)
- Bilinear in dA' and dA
- Symmetric using cyclic property of trace
- Cannot simplify to simple Hessian matrix form (unless vectorized)
Applications of Second Derivatives
- Quadratic approximations: f(x + Δx) ≈ f(x) + F'(x)Δx + ½F''(x)(Δx, Δx)
- Factor of ½ necessary to match second derivative when differentiating approximation
- Higher-order terms O(||Δx||³) dropped
Optimization Applications
- Sequential Quadratic Programming (SQP): approximate nonlinear function by quadratic, constraints by linear
- Newton's method: find root of gradient by linear approximation
- Gradient root-finding: ∇f(x) + F''(x)Δx = 0
Computational Challenges
- For f: ℝⁿ → ℝ, Hessian is n×n matrix
- Huge storage/computation for large n (e.g., billion parameters in neural networks)
- Hessian often approximated rather than computed exactly
Quasi-Newton Methods
- BFGS method (Broyden-Fletcher-Goldfarb-Shanno) - discovered independently by 4 people same year
- DFP method and other related approaches
- Approximate Hessian for large-scale optimization
- Can compute Hessian-vector products efficiently without storing full matrix
