AI summary by TwinMind

Lecture 4: Computational Illumination: dual photography, relighting - Part 1

MIT OpenCourseWare

1h 49m 10s

Course Introduction and Announcements

  • MIT opencourseware content under Creative Commons license
  • Assignment 2 on Optics has two options available
  • Students must check wiki for reading materials not covered in detail (camera mechanics, depth of field, apertures)
  • YouTube tutorials recommended for SLR concepts
  • Final project ideas discussion encouraged
    • Professor Mukawa available as mentor (computer vision leader)
    • Ankit Mohan (previous presenter) also mentoring
    • Maas available for sensor discussions

Previous Class Projects Recognition

  • Sadian photography won best project award
  • Bidirectional screen received student research prize at SIGGRAPH
  • Looking around corner project won Marr prize (top computer vision prize)
  • Kirmani received second best paper award

Homework Requirements Clarification

  • Three required components for assignments:
    • Create personal website with all information
    • Submit link on Stellar
    • Submit input/output photos on Flickr group
  • Only 6 people currently on Flickr group (insufficient participation)
  • Goal: create computationally beautiful photos, not just expensive equipment results
  • Next fascination after HDR and tilt-shift could originate from class techniques

Assignment 2 Options

Option 1: Virtual Optical Bench Extension

  • Extend Andrew Adams' (Stanford PhD) virtual optical bench
  • Built in Flash but can use any language (Java, C++, MATLAB)
  • Allows operations on light as if on optical bench
  • Can place lenses, mirrors, blockers, diffusers
  • Current limitations:
    • Cannot enter specific focal length values (30mm example)
    • No image formation capability
    • Only GUI-based interaction
  • Required additions:
    • More elements (prisms, gratings)
    • Image formation by integrating light from multiple rays
    • Numerical input capability for parameters
  • Programming-intensive option

Option 2: Synthetic Aperture Photography

  • Array of cameras to see behind objects/through foliage
  • Workaround: translate single camera, take 30-40 pictures
  • Same operations as million-dollar camera array
  • Create see-through effects by eliminating foreground objects
  • Physical setup required (Lego robot or manual ruler marking)

Option 3: Software-only Implementation

  • Synthesize images using virtual optical bench
  • OpenGL or virtual optical bench with different viewpoints
  • Less fun but potentially more interesting effects possible

Google Earth Live Discussion

  • Concept: fire up Google Earth, go anywhere, see live video
  • Infrastructure: cameras on street lights, buses, taxis, cell phones
  • Privacy vs commercial applications debate
  • Potential uses:
    • Health indices (walking speed analysis)
    • Real estate (foot traffic analysis)
    • Restaurant wait times
    • Traffic conditions
  • Student poll on timeline: mostly 5-10 years
  • Likely first implementation in non-US city (Hong Kong, Singapore) due to privacy issues
  • Technical challenges: data storage, bandwidth, processing
  • Camera requirements for mobile/street applications

Computational Illumination Theory

Basic Concepts

  • Relationship between lighting (2D) and photos (2D) = 4D problem
  • Can extend to higher dimensions with multiple projectors/cameras
  • Helmholtz reciprocity: can swap light source and eye positions

Dual Photography Mathematics

  • Matrix T: transformation from projector pixels to camera pixels
  • Dimensions: 4 million × 1 million (terabytes of data)
  • T' (transpose) gives dual transformation
  • Sparse vs dense matrices indicate scene complexity
  • Simple scenes: sparse matrices (one-to-one mapping)
  • Complex scenes: dense matrices (multiple reflections)

Practical Implementation

  • Turn on one projector pixel at a time
  • Record full camera image for each
  • Build transformation matrix column by column
  • Computational challenges with large matrices
  • Memory limitations require disk access strategies

Light Transport Analysis

Direct vs Global Illumination

  • Direct reflection: light bounces once
  • Global reflection: light bounces multiple times
  • Subsurface scattering: light enters surface, bounces internally
  • Participating media: fog, water scattering
  • Transmission: light passes through translucent materials

Separation Technique

  • Key insight: global illumination is low-frequency, direct is localized
  • Two-pattern method:
    • Project checkerboard pattern
    • Project inverse checkerboard pattern
    • Global component remains constant
    • Direct component changes with pattern
  • Mathematical solution:
    • I₁ = D + αG (first pattern)
    • I₂ = (1-D) + αG (inverse pattern)
    • Max(I₁,I₂) = direct component
    • Min(I₁,I₂) = global component

Real-World Applications

Material Analysis

  • Marble: mostly subsurface scattering
  • Wax: almost entirely global illumination
  • Glass: specular reflections cause algorithm failures
  • Corner brightening: interreflections accumulate

Practical Examples

  • Shower curtain: see through scattering media
  • Aquarium: clear imaging in murky water
  • Real vs fake fruit detection:
    • Real fruit has internal light bouncing
    • Fake fruit only surface reflection
    • Ripeness detection possible

Skin and Ethnicity

  • Direct component: almost always gray across ethnicities
  • Global component: contains pigmentation
  • Makeup detection possible
  • Professional photography implications

Multi-Projector Systems

  • 6D problem: position (2D) + pixel coordinate (2D) + camera (2D)
  • Reciprocity advantage: cameras work in parallel, projectors sequential
  • Mirror array setup to simulate multiple cameras
  • Reduces capture time from 4 million to 1 million samples

Technical Limitations

  • Controlled lighting environment required
  • Cannot work in bright daylight
  • Specular surfaces cause failures
  • Resolution limited by projector/camera specifications
  • Computational intensity requires overnight processing

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome