AI summary by TwinMind
Lecture 4 Part 2: Nonlinear Root Finding, Optimization, and Adjoint Gradient Methods
MIT OpenCourseWare
44m 26s
Introduction and Course Logistics
- Slides will be posted to GitHub page later today
- Handwritten notes from first part already posted
- Focus on applications of derivative computation and computational methods
Newton's Method for Nonlinear Equations
- Basic use of derivatives: solving nonlinear equations f(x) = 0
- Newton's method: linear algebra technique approximating function by line
- Three-step process:
- Linearize: f(x + ΔX) ≈ f(x) + f'(x)ΔX
- Solve linear problem: set equal to zero, solve for ΔX = -f/f'
- Update: new X = x - f/f'
- Convergence properties:
- Doubles decimal places per iteration when close to root
- Unpredictable when far from root
- Can create fractal patterns (Newton fractals)
Multi-dimensional Newton's Method
- Extends to vector functions: f(x) = 0 where x ∈ ℝⁿ
- n equations, n variables (must match)
- Linearization: f(x + ΔX) ≈ f(x) + f'(x)ΔX
- f'(x) is Jacobian matrix for column vectors
- Solution: ΔX = -J⁻¹f(x) where J is Jacobian
- New iterate: x_new = x - J⁻¹f(x)
- Same convergence properties as 1D case
Optimization Applications
- Scalar function f(x) with vector input x, minimize/maximize
- Machine learning example:
- X = million parameters in neural network
- f(x) = loss function (error between output and target)
- Gradient points uphill, -gradient points downhill (steepest descent)
- Key insight: compute all n derivatives (gradient) with ~same cost as one function evaluation
- Enables simultaneous optimization of all parameters
Optimization Complications
- Step size determination:
- Line search: 1D minimization in gradient direction
- Backtracking: reduce step size if function doesn't improve
- Trust region methods: optimize within bounded region
- Convergence issues:
- Zigzagging in narrow valleys
- Solutions: momentum terms, conjugate gradients, memory methods
- Second-order methods:
- Hessian matrices (second derivatives)
- BFGS algorithm
- No single best algorithm - problem-dependent trial and error
Engineering/Physics Optimization Framework
- Design parameters p (materials, geometry, forces)
- Physical model: solve Ax = B where A, B depend on p
- Solution x represents physics (stresses, fields, etc.)
- Objective function f(x(p)) - scalar performance measure
- Goal: optimize p to minimize/maximize f
- Chain rule required: ∂f/∂p through physics equations
Topology Optimization Example
- Chair design: minimize material while maintaining strength
- Every voxel can be metal or air
- Millions of parameters (density at each voxel)
- Video demonstration: continuous evolution from solid block to optimal structure
- Requires efficient derivative computation for all parameters
Adjoint Methods for Linear Systems
- Problem: f(x) where x solves Ax = B, want ∂f/∂p where p are parameters in A
- Chain rule: df = f'(x) · dx
- Key insight: x = A⁻¹B, so dx = -A⁻¹(∂A/∂p)A⁻¹B
- Full expression: df = -f'(x)A⁻¹(∂A/∂p)x
- Critical optimization: multiply left-to-right
- First: solve A^T v = (f'(x))^T (adjoint equation)
- Then: ∂f/∂p_k = -v^T(∂A/∂p_k)x (dot products)
- Cost: 2 linear solves + cheap dot products vs. million solves for forward mode
Forward Mode vs. Reverse Mode Comparison
- Forward mode (right-to-left chain rule):
- Good for: one input, many outputs
- Bad for many parameters: requires one solve per parameter
- Reverse mode (left-to-right chain rule):
- Good for: many inputs, one output
- Also called: adjoint methods, back propagation
- Only 2 solves regardless of parameter count
- Finite differences equally bad: one solve per parameter perturbation
Adjoint Methods for Nonlinear Systems
- Extension to g(x,p) = 0 nonlinear systems
- Chain rule with implicit function theorem
- Linearization: ∂g/∂x · dx + ∂g/∂p · dp = 0
- Solve for dx: dx = -(∂g/∂x)⁻¹(∂g/∂p)dp
- Adjoint approach: solve (∂g/∂x)^T v = (f'(x))^T
- Final derivatives: ∂f/∂p = -v^T(∂g/∂p)
- Jacobian ∂g/∂x same as used in Newton solver
Practical Implementation Considerations
- Automatic differentiation systems need guidance for implicit equations
- Default AD tries to differentiate through all Newton iterations
- Better approach: tell AD about implicit solve structure
- Physics applications often use legacy code in multiple languages
- Need manual derivatives for parts AD can't handle
- Discretization: often better to differentiate physics before discretizing
