AI summary by TwinMind

David Stork | When Computers Look at Art | Talks at Google

Talks at Google

49m 27s

Introduction and Speaker Background

  • David Stork: founding pioneer in computer vision/ML/AI applied to art history and interpretation
  • Adjunct professor at Stanford, 64 issued patents, 220+ scholarly works, 9 books
  • Published first scholarly works, offered first courses, co-founded first conference in field
  • Author of "Pixels and Paintings" textbook and recent CACM cover article "When Computers Look at Art"

Art Poses Unique Challenges to AI

  • Art museums contain most subtle, interesting, deep, meaningful images ever created
  • Poses problems to scholars for decades/centuries
  • Small worldwide community using computer vision/AI for art history

Key Challenges Not Addressed by Mainstream AI

  • Style analysis: Incredibly important in art vs. traditional photo/video AI
    • Style recognition, quantification, tracking changes over time
    • Image processing in styled works, segmentation, object recognition
  • Small data sets: Most successful AI uses millions/billions of images
    • Artists like Vermeer leave only 34 artworks
    • How to learn from such limited data
  • Non-realistic objects: Soft watches, flying babies with wings
    • Don't appear in normal training sets
    • No physical basis for training
  • Multimodal analysis: X-rays reveal earliest pentimenti (underlying drawings)
    • Shows how artwork changes over time
    • Reveals artist intent, helps detect fakes
  • Unphysical conventions: Artist imagination vs. physics-based photography
  • Abstraction: Major problem in recognizing artists by abstract works
  • Art recovery: Most important artworks lost to fire/flood/war
    • Use surviving sketches, textual descriptions, deep networks
    • Approximate what lost artwork might have looked like
  • Meaning and semantics: Goes beyond surface description to artist's intent/meaning

Pose Analysis in Portraits

  • Traditional art scholars analyze by eye (profile, three-quarters view)
  • Limited to 5-6 artworks at time due to manual process
  • Computer approach: represent angles using roll, yaw, pitch (pilot terminology)
  • Image model: projection of key points (eye corners, nose tip, mouth corners) onto picture plane
  • Two deep neural networks used for inference from key point locations
  • Applied to 11,000 paintings in 5 minutes on laptop

Results Across Art Movements

  • Box whisker diagrams showing roll angle distributions
  • Ukiyo-e (Japanese woodblock prints): Very large average rotation/roll angles
    • Often "mie poses" from Kabuki dances where actors tip heads dramatically
  • Naive/primitive art: Very small rotation angles
    • Untrained artists or deliberate childlike style (e.g., Matisse)
  • System can detect right-handed vs. left-handed self-portraitists across 20,212 portraits
  • Artists tip different ways based on handedness

Lighting Analysis - Girl with Pearl Earring

  • Vermeer widely recognized as master of light rendering
  • Question: What's the direction of illumination?
  • Humans not very good at detecting illumination direction; computer methods superb

Six Methods Used

Model Independent Methods (no 3D face assumptions needed):

  • Cast shadow analysis: Point on occluder (nose tip) to corresponding shadow point = 150° ± 2°
  • Occluding contour algorithm: Based on shape-from-shading computer vision

Model Dependent Methods (require 3D form assumptions):

  • Pearl analysis using computer graphics model
  • Eye highlight analysis using law of reflection
  • Shape-from-shading with computer graphics face model
  • Full 3D computer graphics model

Occluding Contour Algorithm Details

  • Brightness depends on illumination direction, surface normal, albedo, ambient light
  • Problem: Don't know normal vector components (Nx, Ny, Nz)
  • Solution by Nayar and Oren: Along outer boundary, normal perpendicular to contour
  • Z-component = 0 (like flagpole on horizon)
  • Leonardo da Vinci studied this in notebooks (brightness of outer contour vs. light position)
  • Originally developed for forensic photography (detecting photoshopped images)

Results Summary

  • All six methods: 155° ± very small standard deviation
  • Incredible agreement proves careful rendering and actual girl present (not imagination)
  • Can detect multi-figure portraits painted under different lighting conditions

David Hockney Optics Theory Debunking

  • Around 1430: Western painting became dramatically more realistic
  • Examples: Giotto (awkward/schematic) vs. Robert Campin 1430 (photographic realism)
  • Hockney's 2000 theory: Early artists used optics (concave mirrors) during execution
  • Claims based on his own use of photographs as painting reference

Testing Christ in Carpenter's Studio (Georges de La Tour)

  • Hockney claims: Light source outside picture frame, figures painted separately
  • Cast shadow analysis: Shadows point directly to candle, not outside frame
  • Maximum likelihood estimation: Gray contours show highest probability at candle
  • Occluding contour analysis: Red arrows from body contours meet at candle
  • Floor lighting pattern: Physics-based analysis confirms candle position
  • Computer graphics modeling: 19 positions tested, candle location best matches painting

Arnolfini Portrait Analysis

  • Hockney/Falco claim: Convex mirror used as projection mirror
  • Focal length analysis: Painting requires 61 ± 8 cm focal length
  • Mirror focal length: Actual mirror only 18 cm - much too short
  • Perspective analysis: Hockney claimed chandelier "in perfect perspective"
  • Reality: Chandelier lines don't meet at vanishing point, up to 10 cm off
  • Comparison: Modern artist Nicholas Williams painted chandeliers by eye with much better perspective
  • Conclusion: Dozens of peer-reviewed rebuttals confirm Hockney wrong

Advanced Techniques

Segmentation Challenges

  • SegNet and deep neural networks work well on photographs, poorly on paintings
  • Solution: Style transfer to create training data
    • Use deep neural network style transfer (e.g., author + Cézanne painting)
    • Create millions of surrogate images for training
    • Much better segmentation results on actual paintings

Religious Art Analysis

  • Attributes: Objects identifying religious figures (St. Peter's keys to heaven)
  • Deep neural network recognizes different attributes in large painting databases
  • Segmentation finds figures, attribute detection identifies them
  • Color symbolism: Yellow associated with prostitutes (identifying Mary Magdalene)
  • Simple but foundational for computer recognition of religious narratives

Vanitas Paintings

  • Traditional AI recognizes objects (skull, book) but misses deeper meaning
  • Meaning: "Lead humble, sober life to prepare for eternal world"
  • Symbolism: Skull = mortality, watch = time passing, extinguished lamp = life's fragility
  • Approach: Text analysis of 36 vanitas descriptions to extract object-meaning relationships
  • Result: Click on painting objects to get AI explanation of symbolic meaning

Art Recovery Project

  • Example: Gustav Klimt painting burned by Nazis in WWII Vienna
  • Only black and white 1945 photograph survives
  • Colorization: Deep neural network using surviving Klimt works, textual descriptions
  • Art scholars find results very convincing
  • Broader goal: Recover lost masterpieces using sketches, descriptions, surviving works
  • Many most important paintings lost to fire/flood/war

Future Directions and Q&A Insights

Impact on Art Importance

  • Massive data analysis reveals most influential artworks changing art history
  • Only possible with analysis of tens/hundreds of thousands of artworks
  • Lost artwork recovery could change art scholarship entirely
  • Example: "Expulsion of the Moorish Kings" - potentially most important painting ever created, lost to fire

Evolution of AI Tools

  • More data coming online - all Western canon art expected online within 5-10 years
  • Key: Getting art scholars to understand and collaborate with computer scientists
  • Approach: Computer scientists go 95% toward art scholars' interests, not vice versa
  • Tools should become standard like microscopes for biologists, telescopes for astronomers

Unsolved Problems

  • Art recovery: Multi-year project just beginning
  • Semantic meaning: Two visually different paintings conveying same meaning
  • Example: Different global warming warning images (polar bear on ice vs. forest fire) sharing same import
  • Deep problem requiring understanding variations in surface structure vs. underlying meaning

Future of Art Creation vs. Analysis

  • Art will change more in next 10-15 years than last 500 years
  • Focus: Still interested in great masterpieces (oil on canvas) - much to understand
  • Analogy: Computer graphics (creation) vs. computer vision (analysis) - different fields
  • Art studio (creation) vs. art history (analysis) - similar relationship
  • Different notions of success between creation and analysis

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome