AI summary by TwinMind

How does Ray Tracing Work in Video Games and Movies?

Branch Education

29m 21s

Introduction to Ray Tracing in Entertainment

  • Every new TV show/movie with CGI relies on ray tracing for special effects
  • Example: Interstellar battle scenes require 3D artists to model spaceships, position lights, background, camera, then render
  • Rendering = computational process simulating light ray bounces off models
  • Transforms simple 3D models into realistic environments

Path Tracing Algorithm Overview

  • Current industry standard for TV/movies = path tracing
  • Requires unimaginable calculations: entire world population doing 1 calculation/second = 12 days non-stop for single scene
  • Algorithm first conceptualized 1986
  • Took 30 years before movies (Zootopia, Moana, Finding Dory, Coco) could use path tracing
  • Required server farms with thousands of computers + multiple months to complete

3D Scene Construction Process

  • Artists spend months modeling islands, castles, houses, trees, dragons
  • All models broken down into small triangles (GPUs work exclusively with triangles)
  • Example scene: 3.2 million triangles
  • 3D artist assigns textures defining color + material attributes (rough, smooth, metallic, glass, water-like)
  • Models positioned in scene with lights (sky, sun) adjusted for intensity/direction/time of day
  • Virtual camera added for final rendering

Ray Tracing Physics and Implementation

  • Real world: light rays start at sun, hit surfaces, some absorbed/some reflected based on color
  • Tinted rays bounce to camera producing 2D image
  • Near infinite rays produced by sun/sky, only small fraction reach camera
  • Path tracing sends rays FROM camera INTO scene (not from light sources)
  • Determines which objects rays hit, calculates illumination from light sources

Technical Specifications

  • 4K image = 8.3 million pixels
  • Example barbershop scene: 8 million triangles, more complex than island kingdom
  • 8.3 billion rays total (1000 rays per pixel) sent from virtual camera
  • Ray tracing = massively parallel operation (each pixel independent)
  • 1000 rays from one pixel calculated simultaneously with adjacent pixels

Primary Rays and Basic Rendering

  • Primary rays: start at camera, travel through random point in pixel into scene
  • Ray hits triangle → object's color associated with that ray/pixel
  • Example: ray hits red chair → pixel becomes red
  • Nearby rays through same pixel hit close locations, colors averaged together
  • Results in single image with proper 3D perspective from virtual camera
  • Initial image fairly flat-colored (just basic object colors)

Direct Illumination Calculation

  • Determines how surface illuminated directly by light sources
  • From intersection point where primary ray hits triangle, generate shadow rays toward each light source
  • No objects between intersection point and light source = direct illumination
  • Factor in light source brightness, size, color, distance, surface direction
  • All factors multiplied by RGB values of surface material
  • Changes shading/brightness of pixel

Shadow Rays and Shadow Calculation

  • Shadow rays determine if location directly illuminated or in shadow
  • If shadow rays blocked by triangles → no direct illumination → pixel remains dark
  • Backside objects receive no direct illumination from light sources

Indirect Illumination Process

  • Surfaces still have color even in shadows due to light bouncing off walls
  • From primary ray intersection point, generate secondary ray that bounces off
  • Secondary ray hits new surface (e.g., wall point)
  • Send shadow rays from secondary point to each light source
  • Results combined with wall surface color → turns wall point into light source
  • Sometimes requires additional secondary rays if point still in shadows
  • Secondary ray bouncing happens multiple times with shadow rays at each point

Path Tracing Mechanics

  • Purpose: find different paths where light bounces off surfaces, indirectly illuminates original point
  • 1000 rays through random points in single pixel
  • Thousands of secondary rays bounce in different directions
  • Creates accurate approximation for indirect illumination
  • Called "path tracing" because finds billions of paths from camera through scene to light sources

Material Properties and Ray Behavior

  • Secondary ray bounce direction depends on material/texture properties
  • Example: gray spheres with different roughness values
  • Perfectly smooth surface (no roughness) = mirror (perfect reflection direction)
  • 100% roughness = flat gray surface (random bounce directions)
  • Glass material generates refraction rays passing through
  • Pixel color/brightness depends on refraction ray direction and what they hit

Render Settings and Computational Requirements

  • Barbershop scene: 1000 rays per pixel, 4 secondary bounces
  • Different scenes use different numbers of rays/pixel, secondary bounces, light sources
  • Multiplication gives total rays required for single image
  • Animations: 24 frames/second
  • 20-minute animation requires over quadrillion rays
  • Why path tracing considered computationally impossible for decades

Ray-Triangle Intersection Challenges

  • Key problem: determining which triangle out of 8 million each ray hits first
  • Simple method: extend triangle planes, calculate intersection points
  • Check if point inside corresponding triangle
  • Multiple hits require distance calculation for closest triangle
  • Millions of triangles make this incredibly repetitive and computationally problematic

Bounding Volume Hierarchy (BVH) Solution

  • Take triangles, divide into two separate boxes (bounding volumes) using 3D coordinates
  • Each box contains half the triangles
  • Continue dividing: 1.5M → 750K triangles
  • 19 total divisions for 3M triangles
  • Result: 525,000 small boxes at bottom, ~6 triangles each
  • Boxes have sides aligned with coordinate axis for easier calculation

BVH Traversal Process

  • 19 box divide branches form binary tree/hierarchy
  • Each branch: simple ray-box intersection calculation
  • Ray travels to next branch based on which box hit first
  • At bottom: small box with only 6 triangles
  • Perform ray-triangle intersection with just these 6 triangles
  • BVH reduces tens of millions of calculations to handful of simple ray-box intersections + 6 ray-triangle intersections

GPU Hardware Architecture

  • Modern GPU solution for 100+ billion rays per frame
  • Example GPU: 10,496 CUDA/shading cores + 82 ray tracing (RT) cores
  • CUDA cores: basic arithmetic
  • RT cores: specially designed/optimized for ray tracing
  • RT cores have two sections:
    • BVH traversal: takes box coordinates + ray direction, executes BVH traversal in nanoseconds
    • Ray-triangle intersection: uses coordinates of 6 triangles, finds which ray hits first

GPU Performance Comparison

  • RT cores operate in parallel, pipeline operations
  • Few billion rays handled every second
  • Complex scene rendered in 4 minutes
  • 2000 supercomputer (ASCI White): $110M, 12.3 trillion operations/second
  • 2022 NVIDIA 3090 GPU: few thousand dollars, 36 trillion operations/second
  • Incredible computing power fits in shoe box-sized graphics card

Educational Context and Disciplines

  • Ray tracing fusion of multiple disciplines: physics of light, trigonometry, vectors, matrices, computer science algorithms, hardware
  • CGI/special effects previously only for high-budget films, now possible on desktop computers

Video Game Ray Tracing Methods

Method 1: Light Mapping

  • Create very low resolution duplicate of all scene models
  • Use path tracing to determine direct/indirect lighting for low-res objects
  • Save results into light map on low-res duplicate
  • Apply light map to high-res version of objects
  • Creates realistic indirect lighting/shadows on high-res objects
  • Used in Unreal Engine's Lumen renderer

Method 2: Screen Space Ray Tracing

  • Doesn't use scene geometries
  • Uses images/data from video game graphics rendering pipeline
  • 3D transformations build flat 2D image on view screen
  • Additional data created: depth map (distance from camera), normal map (object/pixel directions)
  • Combine view screen + depth map + normal map = approximation of XYZ values + pixel directions
  • Example: reflective lake determines reflection pixels by bouncing rays off lake pixels using normal map
  • Rays continue through simplified geometry, hit trees → reflection of trees on lake
  • Limitation: can only use on-screen data, objects out of view can't be reflected
  • No reflections of objects behind camera
  • Used in games like Cyberpunk

Production Statistics

  • Video creation: 800+ hours of work (research, writing, animation)
  • All animations created using Blender software with path tracing rendering engine
  • Blender = free, powerful modeling/animation software
  • All example scenes downloadable from Blender website

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome