AI summary by TwinMind
Lecture 4: Computational Illumination: dual photography, relighting - Part 1
MIT OpenCourseWare
1h 49m 10s
Course Introduction and Announcements
- MIT opencourseware content under Creative Commons license
- Assignment 2 on Optics has two options available
- Students must check wiki for reading materials not covered in detail (camera mechanics, depth of field, apertures)
- YouTube tutorials recommended for SLR concepts
- Final project ideas discussion encouraged
- Professor Mukawa available as mentor (computer vision leader)
- Ankit Mohan (previous presenter) also mentoring
- Maas available for sensor discussions
Previous Class Projects Recognition
- Sadian photography won best project award
- Bidirectional screen received student research prize at SIGGRAPH
- Looking around corner project won Marr prize (top computer vision prize)
- Kirmani received second best paper award
Homework Requirements Clarification
- Three required components for assignments:
- Create personal website with all information
- Submit link on Stellar
- Submit input/output photos on Flickr group
- Only 6 people currently on Flickr group (insufficient participation)
- Goal: create computationally beautiful photos, not just expensive equipment results
- Next fascination after HDR and tilt-shift could originate from class techniques
Assignment 2 Options
Option 1: Virtual Optical Bench Extension
- Extend Andrew Adams' (Stanford PhD) virtual optical bench
- Built in Flash but can use any language (Java, C++, MATLAB)
- Allows operations on light as if on optical bench
- Can place lenses, mirrors, blockers, diffusers
- Current limitations:
- Cannot enter specific focal length values (30mm example)
- No image formation capability
- Only GUI-based interaction
- Required additions:
- More elements (prisms, gratings)
- Image formation by integrating light from multiple rays
- Numerical input capability for parameters
- Programming-intensive option
Option 2: Synthetic Aperture Photography
- Array of cameras to see behind objects/through foliage
- Workaround: translate single camera, take 30-40 pictures
- Same operations as million-dollar camera array
- Create see-through effects by eliminating foreground objects
- Physical setup required (Lego robot or manual ruler marking)
Option 3: Software-only Implementation
- Synthesize images using virtual optical bench
- OpenGL or virtual optical bench with different viewpoints
- Less fun but potentially more interesting effects possible
Google Earth Live Discussion
- Concept: fire up Google Earth, go anywhere, see live video
- Infrastructure: cameras on street lights, buses, taxis, cell phones
- Privacy vs commercial applications debate
- Potential uses:
- Health indices (walking speed analysis)
- Real estate (foot traffic analysis)
- Restaurant wait times
- Traffic conditions
- Student poll on timeline: mostly 5-10 years
- Likely first implementation in non-US city (Hong Kong, Singapore) due to privacy issues
- Technical challenges: data storage, bandwidth, processing
- Camera requirements for mobile/street applications
Computational Illumination Theory
Basic Concepts
- Relationship between lighting (2D) and photos (2D) = 4D problem
- Can extend to higher dimensions with multiple projectors/cameras
- Helmholtz reciprocity: can swap light source and eye positions
Dual Photography Mathematics
- Matrix T: transformation from projector pixels to camera pixels
- Dimensions: 4 million × 1 million (terabytes of data)
- T' (transpose) gives dual transformation
- Sparse vs dense matrices indicate scene complexity
- Simple scenes: sparse matrices (one-to-one mapping)
- Complex scenes: dense matrices (multiple reflections)
Practical Implementation
- Turn on one projector pixel at a time
- Record full camera image for each
- Build transformation matrix column by column
- Computational challenges with large matrices
- Memory limitations require disk access strategies
Light Transport Analysis
Direct vs Global Illumination
- Direct reflection: light bounces once
- Global reflection: light bounces multiple times
- Subsurface scattering: light enters surface, bounces internally
- Participating media: fog, water scattering
- Transmission: light passes through translucent materials
Separation Technique
- Key insight: global illumination is low-frequency, direct is localized
- Two-pattern method:
- Project checkerboard pattern
- Project inverse checkerboard pattern
- Global component remains constant
- Direct component changes with pattern
- Mathematical solution:
- I₁ = D + αG (first pattern)
- I₂ = (1-D) + αG (inverse pattern)
- Max(I₁,I₂) = direct component
- Min(I₁,I₂) = global component
Real-World Applications
Material Analysis
- Marble: mostly subsurface scattering
- Wax: almost entirely global illumination
- Glass: specular reflections cause algorithm failures
- Corner brightening: interreflections accumulate
Practical Examples
- Shower curtain: see through scattering media
- Aquarium: clear imaging in murky water
- Real vs fake fruit detection:
- Real fruit has internal light bouncing
- Fake fruit only surface reflection
- Ripeness detection possible
Skin and Ethnicity
- Direct component: almost always gray across ethnicities
- Global component: contains pigmentation
- Makeup detection possible
- Professional photography implications
Multi-Projector Systems
- 6D problem: position (2D) + pixel coordinate (2D) + camera (2D)
- Reciprocity advantage: cameras work in parallel, projectors sequential
- Mirror array setup to simulate multiple cameras
- Reduces capture time from 4 million to 1 million samples
Technical Limitations
- Controlled lighting environment required
- Cannot work in bright daylight
- Specular surfaces cause failures
- Resolution limited by projector/camera specifications
- Computational intensity requires overnight processing
