AI summary by TwinMind

Lecture 5 Part 1: Derivative of Matrix Determinant and Inverse

MIT OpenCourseWare

28m 3s

Prerequisites for Defining Derivatives

  • Gradient definition requires inner product to map vector derivatives to scalar functions
  • Derivative definition implicitly requires norm for both input and output spaces
  • Need norm to define "small" terms in little-o notation (terms going to zero faster than ΔX)
  • Formal definition: norm of error term over norm of ΔX → 0 as ΔX → 0
  • Even limit definition requires norm (epsilon-delta proofs need absolute values/norms for vector spaces)
  • Hilbert space: continuous vector space with inner product
  • Banach space: continuous vector space with norm
  • Most cases: define inner product first, get norm for free as square root of inner product with itself

Gradient of Determinant - Formula and Verification

  • Gradient of det(A) = cofactor matrix of A = adjugate of A transpose
  • Multiple equivalent expressions:
    • det(A) · A^(-1) transpose (when A invertible)
    • Adjugate of A transpose
  • Differential form: d(det(A)) = trace(cofactor_transpose · dA)
  • Numerical verification using finite differences with perturbation ~10^(-4) or 10^(-5)
  • Forward mode automatic differentiation (ForwardDiff.jl) matches analytical formula
  • Reverse mode automatic differentiation (Zygote.jl) produces identical results

Mathematical Proofs of Determinant Gradient

Proof 1: Laplace Expansion Method

  • Determinant via Laplace expansion: det(A) = Σ(a_ij · C_ij) where C_ij are cofactors
  • Key insight: each element a_ij appears linearly in exactly one term
  • Derivative ∂det(A)/∂a_ij = coefficient of a_ij = C_ij (the cofactor)
  • Example: 3×3 matrix with symbolic element shows determinant as first-degree polynomial
  • Coefficient of symbolic element equals corresponding minor (4×4 - 3×3 = 13 in example)

Proof 2: Identity Matrix Bootstrap Method

  • Start with characteristic polynomial: det(λI - M) = λ^n + λ^(n-1)·trace(M) + ... + det(M)
  • For small perturbation: det(I + dA) = 1 + trace(dA) + higher order terms
  • General case: det(A + dA) = det(A)·det(I + A^(-1)dA) = det(A)·(1 + trace(A^(-1)dA))
  • Derivative: d(det(A)) = det(A)·trace(A^(-1)dA)
  • Gives same result as cofactor method

Applications and Related Derivatives

Characteristic Polynomial Derivative

  • Characteristic polynomial = product of (x - eigenvalues)
  • Derivative = sum of products taken (n-1) at a time
  • Can be computed directly using determinant gradient formulas

Log Determinant Derivative

  • d(log det(A)) = trace(A^(-1) · dA)
  • Common in optimization (Newton's method: f'/f = d(log f))
  • Logarithmic derivatives and reciprocals appear throughout mathematics

Computational Reality vs Theory

Determinants in Practice

  • Elementary linear algebra: determinants determine singularity (zero = singular)
  • Reality: finite precision makes determinants unreliable for singularity testing
  • Matrix conditioning more important than strict singularity
  • Ill-conditioned: nearly singular
  • Well-conditioned: far from singular
  • Condition numbers (related to singular values) better than determinants for practical assessment

Eigenvalue Computation

  • Textbook method: solve characteristic polynomial roots
  • Reality: characteristic polynomial method not used computationally
  • QR Algorithm: standard method for eigenvalue computation
    • Factor matrix A = QR (orthogonal × upper triangular)
    • Form RQ, repeat: QR → RQ → QR → RQ
    • Eigenvalues appear on diagonal (symmetric case) or in triangular form
    • Related to block power method in multiple dimensions
  • Students typically unaware of actual computational methods beyond power method

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome