AI summary by TwinMind

Lecture 7 Part 2: Second Derivatives, Bilinear Forms, and Hessian Matrices

MIT OpenCourseWare

46m 9s

Introduction to Second Derivatives

  • Second derivatives = derivative of the derivative
  • For functions from matrices to matrices, need careful consideration of what second derivative represents
  • First derivative F'(x) takes small change dx, outputs df (first-order approximation)
  • Second derivative naturally defined as derivative of F'

Linear Operators and Vector Spaces

  • F'(x + dx') - F'(x) represents difference of two linear operators
  • Linear operators form vector space
  • Addition/subtraction of operators: (L1 + L2)(v) = L1(v) + L2(v)
  • Scalar multiplication: (αL1)(v) = α(L1(v))
  • Matrix operations derive from these linear operator properties

Bilinear Forms

  • Second derivative F''(x) takes two inputs: dx' and dx
  • F''(x)(dx', dx) = F'(x + dx') - F'(x)
  • Bilinear form: linear in both arguments
  • Properties: B(u1 + u2, v) = B(u1, v) + B(u2, v) and B(u, v1 + v2) = B(u, v1) + B(u, v2)
  • Scaling both arguments: B(2u, 3v) = 6B(u, v)

Symmetry of Second Derivatives

  • F''(x)(dx', dx) = F''(x)(dx, dx')
  • Proof by expanding definition using commutativity of vector addition
  • Writing out F'(x + dx')(dx) - F'(x)(dx) shows both dx and dx' appear symmetrically
  • This generalizes the familiar property from single-variable calculus

Multivariable Calculus Example

  • Scalar function f: ℝⁿ → ℝ
  • F'(x) is row vector (transpose of gradient)
  • F''(x) takes two vectors, outputs scalar
  • Only way to do this: dx'ᵀ H dx where H is n×n matrix
  • Must be symmetric matrix due to symmetry property
  • H called Hessian matrix

Hessian Matrix Components

  • Gradient ∇f = [∂f/∂x₁, ..., ∂f/∂xₙ]ᵀ
  • D(∇f) = H dx where H is Jacobian of gradient
  • Hij = ∂²f/∂xi∂xj
  • Symmetry: Hij = Hji (can swap order of partial derivatives)
  • This familiar 1802 property comes from general derivative definition

Matrix Determinant Example

  • f(A) = det(A) where A is matrix
  • F'(A) = det(A) · trace(A⁻¹ dA)
  • Second derivative calculation using product rule and chain rule
  • F''(A)(dA', dA) = det(A) · trace(A⁻¹ dA') · trace(A⁻¹ dA) - det(A) · trace(A⁻¹ dA' A⁻¹ dA)
  • Bilinear in dA' and dA
  • Symmetric using cyclic property of trace
  • Cannot simplify to simple Hessian matrix form (unless vectorized)

Applications of Second Derivatives

  • Quadratic approximations: f(x + Δx) ≈ f(x) + F'(x)Δx + ½F''(x)(Δx, Δx)
  • Factor of ½ necessary to match second derivative when differentiating approximation
  • Higher-order terms O(||Δx||³) dropped

Optimization Applications

  • Sequential Quadratic Programming (SQP): approximate nonlinear function by quadratic, constraints by linear
  • Newton's method: find root of gradient by linear approximation
  • Gradient root-finding: ∇f(x) + F''(x)Δx = 0

Computational Challenges

  • For f: ℝⁿ → ℝ, Hessian is n×n matrix
  • Huge storage/computation for large n (e.g., billion parameters in neural networks)
  • Hessian often approximated rather than computed exactly

Quasi-Newton Methods

  • BFGS method (Broyden-Fletcher-Goldfarb-Shanno) - discovered independently by 4 people same year
  • DFP method and other related approaches
  • Approximate Hessian for large-scale optimization
  • Can compute Hessian-vector products efficiently without storing full matrix

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome