AI summary by TwinMind
Lecture 5 Part 1: Derivative of Matrix Determinant and Inverse
MIT OpenCourseWare
28m 3s
Prerequisites for Defining Derivatives
- Gradient definition requires inner product to map vector derivatives to scalar functions
- Derivative definition implicitly requires norm for both input and output spaces
- Need norm to define "small" terms in little-o notation (terms going to zero faster than ΔX)
- Formal definition: norm of error term over norm of ΔX → 0 as ΔX → 0
- Even limit definition requires norm (epsilon-delta proofs need absolute values/norms for vector spaces)
- Hilbert space: continuous vector space with inner product
- Banach space: continuous vector space with norm
- Most cases: define inner product first, get norm for free as square root of inner product with itself
Gradient of Determinant - Formula and Verification
- Gradient of det(A) = cofactor matrix of A = adjugate of A transpose
- Multiple equivalent expressions:
- det(A) · A^(-1) transpose (when A invertible)
- Adjugate of A transpose
- Differential form: d(det(A)) = trace(cofactor_transpose · dA)
- Numerical verification using finite differences with perturbation ~10^(-4) or 10^(-5)
- Forward mode automatic differentiation (ForwardDiff.jl) matches analytical formula
- Reverse mode automatic differentiation (Zygote.jl) produces identical results
Mathematical Proofs of Determinant Gradient
Proof 1: Laplace Expansion Method
- Determinant via Laplace expansion: det(A) = Σ(a_ij · C_ij) where C_ij are cofactors
- Key insight: each element a_ij appears linearly in exactly one term
- Derivative ∂det(A)/∂a_ij = coefficient of a_ij = C_ij (the cofactor)
- Example: 3×3 matrix with symbolic element shows determinant as first-degree polynomial
- Coefficient of symbolic element equals corresponding minor (4×4 - 3×3 = 13 in example)
Proof 2: Identity Matrix Bootstrap Method
- Start with characteristic polynomial: det(λI - M) = λ^n + λ^(n-1)·trace(M) + ... + det(M)
- For small perturbation: det(I + dA) = 1 + trace(dA) + higher order terms
- General case: det(A + dA) = det(A)·det(I + A^(-1)dA) = det(A)·(1 + trace(A^(-1)dA))
- Derivative: d(det(A)) = det(A)·trace(A^(-1)dA)
- Gives same result as cofactor method
Applications and Related Derivatives
Characteristic Polynomial Derivative
- Characteristic polynomial = product of (x - eigenvalues)
- Derivative = sum of products taken (n-1) at a time
- Can be computed directly using determinant gradient formulas
Log Determinant Derivative
- d(log det(A)) = trace(A^(-1) · dA)
- Common in optimization (Newton's method: f'/f = d(log f))
- Logarithmic derivatives and reciprocals appear throughout mathematics
Computational Reality vs Theory
Determinants in Practice
- Elementary linear algebra: determinants determine singularity (zero = singular)
- Reality: finite precision makes determinants unreliable for singularity testing
- Matrix conditioning more important than strict singularity
- Ill-conditioned: nearly singular
- Well-conditioned: far from singular
- Condition numbers (related to singular values) better than determinants for practical assessment
Eigenvalue Computation
- Textbook method: solve characteristic polynomial roots
- Reality: characteristic polynomial method not used computationally
- QR Algorithm: standard method for eigenvalue computation
- Factor matrix A = QR (orthogonal × upper triangular)
- Form RQ, repeat: QR → RQ → QR → RQ
- Eigenvalues appear on diagonal (symmetric case) or in triangular form
- Related to block power method in multiple dimensions
- Students typically unaware of actual computational methods beyond power method
