AI summary by TwinMind
David Stork | When Computers Look at Art | Talks at Google
Talks at Google
49m 27s
Introduction and Speaker Background
- David Stork: founding pioneer in computer vision/ML/AI applied to art history and interpretation
- Adjunct professor at Stanford, 64 issued patents, 220+ scholarly works, 9 books
- Published first scholarly works, offered first courses, co-founded first conference in field
- Author of "Pixels and Paintings" textbook and recent CACM cover article "When Computers Look at Art"
Art Poses Unique Challenges to AI
- Art museums contain most subtle, interesting, deep, meaningful images ever created
- Poses problems to scholars for decades/centuries
- Small worldwide community using computer vision/AI for art history
Key Challenges Not Addressed by Mainstream AI
- Style analysis: Incredibly important in art vs. traditional photo/video AI
- Style recognition, quantification, tracking changes over time
- Image processing in styled works, segmentation, object recognition
- Small data sets: Most successful AI uses millions/billions of images
- Artists like Vermeer leave only 34 artworks
- How to learn from such limited data
- Non-realistic objects: Soft watches, flying babies with wings
- Don't appear in normal training sets
- No physical basis for training
- Multimodal analysis: X-rays reveal earliest pentimenti (underlying drawings)
- Shows how artwork changes over time
- Reveals artist intent, helps detect fakes
- Unphysical conventions: Artist imagination vs. physics-based photography
- Abstraction: Major problem in recognizing artists by abstract works
- Art recovery: Most important artworks lost to fire/flood/war
- Use surviving sketches, textual descriptions, deep networks
- Approximate what lost artwork might have looked like
- Meaning and semantics: Goes beyond surface description to artist's intent/meaning
Pose Analysis in Portraits
- Traditional art scholars analyze by eye (profile, three-quarters view)
- Limited to 5-6 artworks at time due to manual process
- Computer approach: represent angles using roll, yaw, pitch (pilot terminology)
- Image model: projection of key points (eye corners, nose tip, mouth corners) onto picture plane
- Two deep neural networks used for inference from key point locations
- Applied to 11,000 paintings in 5 minutes on laptop
Results Across Art Movements
- Box whisker diagrams showing roll angle distributions
- Ukiyo-e (Japanese woodblock prints): Very large average rotation/roll angles
- Often "mie poses" from Kabuki dances where actors tip heads dramatically
- Naive/primitive art: Very small rotation angles
- Untrained artists or deliberate childlike style (e.g., Matisse)
- System can detect right-handed vs. left-handed self-portraitists across 20,212 portraits
- Artists tip different ways based on handedness
Lighting Analysis - Girl with Pearl Earring
- Vermeer widely recognized as master of light rendering
- Question: What's the direction of illumination?
- Humans not very good at detecting illumination direction; computer methods superb
Six Methods Used
Model Independent Methods (no 3D face assumptions needed):
- Cast shadow analysis: Point on occluder (nose tip) to corresponding shadow point = 150° ± 2°
- Occluding contour algorithm: Based on shape-from-shading computer vision
Model Dependent Methods (require 3D form assumptions):
- Pearl analysis using computer graphics model
- Eye highlight analysis using law of reflection
- Shape-from-shading with computer graphics face model
- Full 3D computer graphics model
Occluding Contour Algorithm Details
- Brightness depends on illumination direction, surface normal, albedo, ambient light
- Problem: Don't know normal vector components (Nx, Ny, Nz)
- Solution by Nayar and Oren: Along outer boundary, normal perpendicular to contour
- Z-component = 0 (like flagpole on horizon)
- Leonardo da Vinci studied this in notebooks (brightness of outer contour vs. light position)
- Originally developed for forensic photography (detecting photoshopped images)
Results Summary
- All six methods: 155° ± very small standard deviation
- Incredible agreement proves careful rendering and actual girl present (not imagination)
- Can detect multi-figure portraits painted under different lighting conditions
David Hockney Optics Theory Debunking
- Around 1430: Western painting became dramatically more realistic
- Examples: Giotto (awkward/schematic) vs. Robert Campin 1430 (photographic realism)
- Hockney's 2000 theory: Early artists used optics (concave mirrors) during execution
- Claims based on his own use of photographs as painting reference
Testing Christ in Carpenter's Studio (Georges de La Tour)
- Hockney claims: Light source outside picture frame, figures painted separately
- Cast shadow analysis: Shadows point directly to candle, not outside frame
- Maximum likelihood estimation: Gray contours show highest probability at candle
- Occluding contour analysis: Red arrows from body contours meet at candle
- Floor lighting pattern: Physics-based analysis confirms candle position
- Computer graphics modeling: 19 positions tested, candle location best matches painting
Arnolfini Portrait Analysis
- Hockney/Falco claim: Convex mirror used as projection mirror
- Focal length analysis: Painting requires 61 ± 8 cm focal length
- Mirror focal length: Actual mirror only 18 cm - much too short
- Perspective analysis: Hockney claimed chandelier "in perfect perspective"
- Reality: Chandelier lines don't meet at vanishing point, up to 10 cm off
- Comparison: Modern artist Nicholas Williams painted chandeliers by eye with much better perspective
- Conclusion: Dozens of peer-reviewed rebuttals confirm Hockney wrong
Advanced Techniques
Segmentation Challenges
- SegNet and deep neural networks work well on photographs, poorly on paintings
- Solution: Style transfer to create training data
- Use deep neural network style transfer (e.g., author + Cézanne painting)
- Create millions of surrogate images for training
- Much better segmentation results on actual paintings
Religious Art Analysis
- Attributes: Objects identifying religious figures (St. Peter's keys to heaven)
- Deep neural network recognizes different attributes in large painting databases
- Segmentation finds figures, attribute detection identifies them
- Color symbolism: Yellow associated with prostitutes (identifying Mary Magdalene)
- Simple but foundational for computer recognition of religious narratives
Vanitas Paintings
- Traditional AI recognizes objects (skull, book) but misses deeper meaning
- Meaning: "Lead humble, sober life to prepare for eternal world"
- Symbolism: Skull = mortality, watch = time passing, extinguished lamp = life's fragility
- Approach: Text analysis of 36 vanitas descriptions to extract object-meaning relationships
- Result: Click on painting objects to get AI explanation of symbolic meaning
Art Recovery Project
- Example: Gustav Klimt painting burned by Nazis in WWII Vienna
- Only black and white 1945 photograph survives
- Colorization: Deep neural network using surviving Klimt works, textual descriptions
- Art scholars find results very convincing
- Broader goal: Recover lost masterpieces using sketches, descriptions, surviving works
- Many most important paintings lost to fire/flood/war
Future Directions and Q&A Insights
Impact on Art Importance
- Massive data analysis reveals most influential artworks changing art history
- Only possible with analysis of tens/hundreds of thousands of artworks
- Lost artwork recovery could change art scholarship entirely
- Example: "Expulsion of the Moorish Kings" - potentially most important painting ever created, lost to fire
Evolution of AI Tools
- More data coming online - all Western canon art expected online within 5-10 years
- Key: Getting art scholars to understand and collaborate with computer scientists
- Approach: Computer scientists go 95% toward art scholars' interests, not vice versa
- Tools should become standard like microscopes for biologists, telescopes for astronomers
Unsolved Problems
- Art recovery: Multi-year project just beginning
- Semantic meaning: Two visually different paintings conveying same meaning
- Example: Different global warming warning images (polar bear on ice vs. forest fire) sharing same import
- Deep problem requiring understanding variations in surface structure vs. underlying meaning
Future of Art Creation vs. Analysis
- Art will change more in next 10-15 years than last 500 years
- Focus: Still interested in great masterpieces (oil on canvas) - much to understand
- Analogy: Computer graphics (creation) vs. computer vision (analysis) - different fields
- Art studio (creation) vs. art history (analysis) - similar relationship
- Different notions of success between creation and analysis
