AI summary by TwinMind
Alvy Ray Smith | A Biography of the Pixel | Talks at Google
Talks at Google
56m 6s
Introduction and Book Overview
- Dr. Alvy Ray Smith, Pixar co-founder and computer graphics pioneer, discusses his book "A Biography of the Pixel"
- Smith: first director of computer graphics at Lucasfilm, first graphics fellow at Microsoft, two Technical Academy Awards
- Book examines pixel technology from multiple angles: art, technology, entertainment, business, history
- Story spans from Fourier waves through Turing machines to first digital movies from Pixar, DreamWorks, Blue Sky
What is a Pixel - Fundamental Definition
- Common misconception: pixels are NOT little abutting squares
- Correct definition: pixel = sample according to sampling theorem at particular frequency representing a picture
- Picture viewed as two-dimensional signal sampled at specific rate defined by sampling theorem
- Can discard infinity of points in signal, maintain only samples (pixels) without losing picture information
- Reconstruction possible: throw away infinity, not lose your picture - "mindblower" theorem
Sampling Theorem History and Attribution
- Commonly attributed to Claude Shannon, but Smith challenges this attribution
- Shannon never claimed he proved it first, just provided details of existing knowledge
- Smith credits Vladimir Kotelnikov, Russian communist who proved it in 1933
- Suggests American reluctance to credit communist for important discovery
- Kotelnikov's theorem: sample at regular intervals in 2D, can reconstruct original signal perfectly
Fourier Theory and Wave Mathematics
- Fourier (around 1800): all pictures are sums of regular waves (different frequencies/amplitudes)
- Two-dimensional Fourier waves look like corrugations with sine wave cross-sections
- Any picture (TV images, personal photos, Milky Way) = sum of regular waves
- Fourier also discovered greenhouse effect around 1800, involved with Napoleon and Rosetta Stone
- Modern media world based on this "mindblower" - Zoom, nearly all pictures now digital
Sampling Rate and Reconstruction
- Magic frequency: sample at twice the highest frequency in the sum (Kotelnikov's rule)
- Whole modern media world based on this principle
- Perfect reconstruction possible "10 years later on other side of planet"
- Display devices have array of "display elements" (not pixels) - analog blobs of light
- Pixels (invisible samples) spread by display elements to reconstruct original picture
Theory vs. Engineering Reality
- Ideal theory: spread pixels using "sinc" function (ripples to infinity)
- Engineering reality: can't handle infinite width, use finite approximation
- Science vs. engineering dichotomy throughout technology development
- Works well despite approximations
Early Cinema and Naive Sampling
- Movie pioneers (1895) approached sampling "completely naively" - no understanding of theory
- Movies sample in time using "frames" - one after another
- Current system: project individual frames with blackness between, brain creates motion
- Regular movies show each frame twice (not true sampling theory)
- Animation: sometimes each frame shown four times ("shooting on twos" - 12 fps, each frame hits retina 4 times)
Ideal Movie System Theory
- Smith proposes sampling theorem-based movie system (never built)
- Would sample visual flow at twice highest frequency
- Reconstruction: several frames overlap in time, add together
- Result: ideal reconstruction of visual flow before entering pupil
- Would see "little bit of few frames ago, little bit of one frame ago, current frame, future frames"
Animation Language and Technical Workarounds
- Technical limitations forced animators to develop visual language not true to reality
- Speed lines, puffs, anticipation, squash and stretch, exaggeration
- These became "language to the brain" - clues for proper interpretation
- Smith's hand-clapping animation experiment: realistic version boring, exaggerated version fun
- Technical constraints created artistic language that persists even when constraints removed
Moore's Law Impact
- Gordon Moore announced law in 1965
- Smith's intuitive version: "everything good about computers gets better by order of magnitude every five years"
- Order of magnitude = factor of 10, about limit of human conceptual change
- From 1965 to present: 100 billion x improvement factor
- Enables reconceptualizations every five years
- Moore's Law still continuing despite frequent death announcements
- IBM announced 2 nanometer technology - approaching trillion x improvement
Early Pixar and Moore's Law Strategy
- 1975 at New York Institute of Technology: conceived first digital movie goal
- Didn't understand would take 20 years
- Smith and Ed Catmull calculated costs on envelopes: 1975 = several billion dollars, decade timeline
- Strategy: assume Moore's Law holds, prepare technologies and algorithms
- Competitor Gary Demos/John Whitney Jr. bought Cray supercomputer
- Smith calculated: economically unfeasible unless everyone signs up for Cray time (they didn't)
Pixar's Technical-Artistic Culture
- Deliberate choice to stress equity across disciplines
- Smith: artist background (learned oil painting from Uncle George) + computer expertise
- Disallowed looking down on either artists or technologists
- Required mutual admiration and dignity assignment
- Business required both technically creative and artistically creative people working together
- Counter-examples: graphics houses looking down on "technoids," Microsoft treating art as lesser
Jam Sessions and Collaborative Development
- Experimental overnight sessions between technical and artistic people
- Smith and David DiFrancisco example: alternated on single paint tablet
- "No idea where paintings were going" - mostly junk but fun
- Artists key to designing software interfaces
- Iterative process: "scratch heads when they ask for things, then do it and say 'oh that's what they meant'"
Finding Nemo Technical Restraint Example
- Originally created underwater world as realistic as real underwater world
- Had to "back it off" because gorgeous visuals took attention from characters
- Deliberate decision: story and character animation first, technology as tool
- "Don't make it quite so good" - blur it down
Recommendations for Future Creators
- SIGGRAPH as primary recommendation for 40+ years
- Annual conference where all industry players show off to peers
- At height: 40,000 attendees yearly
- Technical papers, artistic productions, art shows, educational content
- "Rich environment" for meeting key people and seeing cutting edge
Andre and Wally B at SIGGRAPH 1984
- Demonstrated solved motion blur problem
- John Lasseter's first work with group (later Pixar)
- 5,000 SIGGRAPH attendees understood significance and "screamed heads off"
- George Lucas attended (visiting girlfriend) but didn't understand breakthrough
- Lucas saw crude aspects, SIGGRAPH audience filled in blanks with Moore's Law projection
- Good management: Lucas didn't share disappointment with team
The Movie Goal and Toy Story Path
- Goal since 1975: first completely digital movie in world
- Didn't know format (2D/3D) or content, just completely computer-generated
- Path that worked: new 3D modeling approach nobody had seen
- "Flipped everybody out" and changed moviemaking world
- Five years as hardware company waiting for Moore's Law to enable movie production
- Japanese Monkey King project calculations showed still too expensive/time-consuming
Historical Research Challenges
- Surprised by "lousy histories of technologies" - expected to "look it up and write it down"
- Every looked-up source was wrong
- Examples: sampling theorem attribution, movie invention, television invention
- Avoided "simple narrative" of single genius having idea
- Developed flowcharts/genealogies showing complex interactions, borrowing, stealing
- "Much more satisfying" and truthful approach to technological history
Audience Q&A Highlights
- Windup toys question: directed to John Lasseter as actual collector
- Screen element limits: Moore's Law doesn't apply to display elements (have physical size requirements)
- AI in entertainment: Smith's 1965 Stanford AI experience, changed to achievable goals
- Recent AI developments impressive but maintains skepticism due to marketing history
- "Zebras and horses" paper example: AI solving problems too complex for human understanding
- Definition: "if you can understand it, it's not going to be intelligent"
