AI summary by TwinMind
How does Ray Tracing Work in Video Games and Movies?
Branch Education
29m 21s
Introduction to Ray Tracing in Entertainment
- Every new TV show/movie with CGI relies on ray tracing for special effects
- Example: Interstellar battle scenes require 3D artists to model spaceships, position lights, background, camera, then render
- Rendering = computational process simulating light ray bounces off models
- Transforms simple 3D models into realistic environments
Path Tracing Algorithm Overview
- Current industry standard for TV/movies = path tracing
- Requires unimaginable calculations: entire world population doing 1 calculation/second = 12 days non-stop for single scene
- Algorithm first conceptualized 1986
- Took 30 years before movies (Zootopia, Moana, Finding Dory, Coco) could use path tracing
- Required server farms with thousands of computers + multiple months to complete
3D Scene Construction Process
- Artists spend months modeling islands, castles, houses, trees, dragons
- All models broken down into small triangles (GPUs work exclusively with triangles)
- Example scene: 3.2 million triangles
- 3D artist assigns textures defining color + material attributes (rough, smooth, metallic, glass, water-like)
- Models positioned in scene with lights (sky, sun) adjusted for intensity/direction/time of day
- Virtual camera added for final rendering
Ray Tracing Physics and Implementation
- Real world: light rays start at sun, hit surfaces, some absorbed/some reflected based on color
- Tinted rays bounce to camera producing 2D image
- Near infinite rays produced by sun/sky, only small fraction reach camera
- Path tracing sends rays FROM camera INTO scene (not from light sources)
- Determines which objects rays hit, calculates illumination from light sources
Technical Specifications
- 4K image = 8.3 million pixels
- Example barbershop scene: 8 million triangles, more complex than island kingdom
- 8.3 billion rays total (1000 rays per pixel) sent from virtual camera
- Ray tracing = massively parallel operation (each pixel independent)
- 1000 rays from one pixel calculated simultaneously with adjacent pixels
Primary Rays and Basic Rendering
- Primary rays: start at camera, travel through random point in pixel into scene
- Ray hits triangle → object's color associated with that ray/pixel
- Example: ray hits red chair → pixel becomes red
- Nearby rays through same pixel hit close locations, colors averaged together
- Results in single image with proper 3D perspective from virtual camera
- Initial image fairly flat-colored (just basic object colors)
Direct Illumination Calculation
- Determines how surface illuminated directly by light sources
- From intersection point where primary ray hits triangle, generate shadow rays toward each light source
- No objects between intersection point and light source = direct illumination
- Factor in light source brightness, size, color, distance, surface direction
- All factors multiplied by RGB values of surface material
- Changes shading/brightness of pixel
Shadow Rays and Shadow Calculation
- Shadow rays determine if location directly illuminated or in shadow
- If shadow rays blocked by triangles → no direct illumination → pixel remains dark
- Backside objects receive no direct illumination from light sources
Indirect Illumination Process
- Surfaces still have color even in shadows due to light bouncing off walls
- From primary ray intersection point, generate secondary ray that bounces off
- Secondary ray hits new surface (e.g., wall point)
- Send shadow rays from secondary point to each light source
- Results combined with wall surface color → turns wall point into light source
- Sometimes requires additional secondary rays if point still in shadows
- Secondary ray bouncing happens multiple times with shadow rays at each point
Path Tracing Mechanics
- Purpose: find different paths where light bounces off surfaces, indirectly illuminates original point
- 1000 rays through random points in single pixel
- Thousands of secondary rays bounce in different directions
- Creates accurate approximation for indirect illumination
- Called "path tracing" because finds billions of paths from camera through scene to light sources
Material Properties and Ray Behavior
- Secondary ray bounce direction depends on material/texture properties
- Example: gray spheres with different roughness values
- Perfectly smooth surface (no roughness) = mirror (perfect reflection direction)
- 100% roughness = flat gray surface (random bounce directions)
- Glass material generates refraction rays passing through
- Pixel color/brightness depends on refraction ray direction and what they hit
Render Settings and Computational Requirements
- Barbershop scene: 1000 rays per pixel, 4 secondary bounces
- Different scenes use different numbers of rays/pixel, secondary bounces, light sources
- Multiplication gives total rays required for single image
- Animations: 24 frames/second
- 20-minute animation requires over quadrillion rays
- Why path tracing considered computationally impossible for decades
Ray-Triangle Intersection Challenges
- Key problem: determining which triangle out of 8 million each ray hits first
- Simple method: extend triangle planes, calculate intersection points
- Check if point inside corresponding triangle
- Multiple hits require distance calculation for closest triangle
- Millions of triangles make this incredibly repetitive and computationally problematic
Bounding Volume Hierarchy (BVH) Solution
- Take triangles, divide into two separate boxes (bounding volumes) using 3D coordinates
- Each box contains half the triangles
- Continue dividing: 1.5M → 750K triangles
- 19 total divisions for 3M triangles
- Result: 525,000 small boxes at bottom, ~6 triangles each
- Boxes have sides aligned with coordinate axis for easier calculation
BVH Traversal Process
- 19 box divide branches form binary tree/hierarchy
- Each branch: simple ray-box intersection calculation
- Ray travels to next branch based on which box hit first
- At bottom: small box with only 6 triangles
- Perform ray-triangle intersection with just these 6 triangles
- BVH reduces tens of millions of calculations to handful of simple ray-box intersections + 6 ray-triangle intersections
GPU Hardware Architecture
- Modern GPU solution for 100+ billion rays per frame
- Example GPU: 10,496 CUDA/shading cores + 82 ray tracing (RT) cores
- CUDA cores: basic arithmetic
- RT cores: specially designed/optimized for ray tracing
- RT cores have two sections:
- BVH traversal: takes box coordinates + ray direction, executes BVH traversal in nanoseconds
- Ray-triangle intersection: uses coordinates of 6 triangles, finds which ray hits first
GPU Performance Comparison
- RT cores operate in parallel, pipeline operations
- Few billion rays handled every second
- Complex scene rendered in 4 minutes
- 2000 supercomputer (ASCI White): $110M, 12.3 trillion operations/second
- 2022 NVIDIA 3090 GPU: few thousand dollars, 36 trillion operations/second
- Incredible computing power fits in shoe box-sized graphics card
Educational Context and Disciplines
- Ray tracing fusion of multiple disciplines: physics of light, trigonometry, vectors, matrices, computer science algorithms, hardware
- CGI/special effects previously only for high-budget films, now possible on desktop computers
Video Game Ray Tracing Methods
Method 1: Light Mapping
- Create very low resolution duplicate of all scene models
- Use path tracing to determine direct/indirect lighting for low-res objects
- Save results into light map on low-res duplicate
- Apply light map to high-res version of objects
- Creates realistic indirect lighting/shadows on high-res objects
- Used in Unreal Engine's Lumen renderer
Method 2: Screen Space Ray Tracing
- Doesn't use scene geometries
- Uses images/data from video game graphics rendering pipeline
- 3D transformations build flat 2D image on view screen
- Additional data created: depth map (distance from camera), normal map (object/pixel directions)
- Combine view screen + depth map + normal map = approximation of XYZ values + pixel directions
- Example: reflective lake determines reflection pixels by bouncing rays off lake pixels using normal map
- Rays continue through simplified geometry, hit trees → reflection of trees on lake
- Limitation: can only use on-screen data, objects out of view can't be reflected
- No reflections of objects behind camera
- Used in games like Cyberpunk
Production Statistics
- Video creation: 800+ hours of work (research, writing, animation)
- All animations created using Blender software with path tracing rendering engine
- Blender = free, powerful modeling/animation software
- All example scenes downloadable from Blender website
