AI summary by TwinMind

How do Video Game Graphics Work?

Branch Education

21m 0s

Video Game Graphics Complexity

  • Modern video games transport players to detailed environments using spectacular graphics
  • Red Dead Redemption 2 train station example:
    • 2.1 million vertices assembled into 3.5 million triangles
    • 976 colors and textures assigned to surfaces
    • Virtual sun illumination
    • All composed entirely of ones and zeros processed by graphics card/console

Graphics Rendering Pipeline Overview

  • Three core steps used for decades in thousands of games:
    • Vertex shading
    • Rasterization
    • Fragment shading
  • Additional steps used in modern games but these three remain backbone
  • Algorithm foundation for computer and console games

Vertex Shading Process

  • Takes 3D objects/geometries/meshes and calculates placement in 2D view screen
  • Uses camera field of view to determine visible objects
  • Train station scene example:
    • 1,100 different models total
    • Camera field of view reduces to 600 objects needing rendering
  • Locomotive breakdown:
    • 762,000 flat triangles
    • 382,000 vertices
    • 9 different materials/colors
  • Three transformation stages:
    • Model space to world space
    • World space to camera space
    • Perspective field of view to view screen
  • Uses XYZ coordinates, position/scale/rotation, camera coordinates/rotation
  • Matrix multiplication results in X/Y values on view screen plus Z depth value
  • All 2.1 million vertices of 600 objects undergo similar transformations
  • 3.5 million triangles moved onto 2D view screen

GPU Processing Power

  • Incredible amount of matrix math handled by specialized GPU cores
  • Example GPU specifications:
    • 10,000+ cores designed for triangle mesh rendering
    • 35 trillion operations of 32-bit multiplication/addition per second
    • Vertex coordinates/transformation data distributed among cores
    • Results in 120+ frames per second rendering

Rasterization Process

  • Determines which pixels covered by each triangle
  • 4K monitor resolution: 3840 x 2160 = 8.3 million pixels
  • GPU calculates triangle placement within pixel grid
  • Pixels shaded using triangle's assigned texture/color
  • Triangles converted to fragments (groups of pixels from same triangle)
  • Process repeated for all 3.5 million triangles
  • RGB color values applied to appropriate pixels
  • 4K image formed in frame buffer and sent to display

Depth/Visibility Handling (Z-Buffer)

  • Solves visibility problem for overlapping/blocking triangles
  • Z-buffer adds depth value to each of 8.3 million pixels
  • Corresponds to distance from camera
  • Triangle rasterization process:
    • Compare triangle's Z values with Z-buffer values
    • Lower Z values (closer to camera) → paint pixels and update Z-buffer
    • Higher Z values (further away) → discard triangle
  • Only closest triangles with lowest Z values displayed
  • Individual pixels compute Z values using vertex coordinates
  • Enables proper intersection rendering pixel by pixel

Anti-Aliasing (SSAA)

  • Addresses jagged/pixelated edges from angled triangle cuts
  • Super Sampling Anti-Aliasing technique:
    • 16 sampling points distributed across single pixel
    • Fractional shade applied based on sampling points covered
    • Results in faded edges and reduced pixelation

Real-Time Processing Requirements

  • Camera view and objects continuously moving during gameplay
  • Vertex shading, rasterization, fragment shading recalculated every frame
  • 120 FPS = recalculation every 8.3 milliseconds

Fragment Shading Purpose

  • Beyond simple color application - creates realistic scenes
  • Accounts for:
    • Light direction and strength
    • Camera position
    • Reflections
    • Shadows from other objects
  • Shades pixels with accurate illumination

Fragment Shading Examples

  • Train engine demonstration:
    • Black metal base color → horribly inaccurate without shading
    • Proper shading: darker bottom, lighter top, specular highlights
    • Results in realistic black metal appearance
  • Dynamic lighting effects:
    • Sun movement reflects time passage
    • Night scenes: darker materials, fire illumination
  • Historical example: Super Mario 64 (30 years old) used simple shading

Surface Normal Calculations

  • Surface brightness depends on angle relative to light source
  • Direct facing → brighter shading
  • Perpendicular/away facing → darker shading
  • Two key requirements:
    • Light direction
    • Triangle surface direction (surface normal)
  • Locomotive example: 762,000 triangles facing different directions
  • Surface normal = direction perpendicular to triangle plane

Lighting Mathematics

  • Cosine of angle (theta) between light direction and surface normal
  • Cosine theta values:
    • 1 when surface faces light
    • 0 when perpendicular to light
  • Final color calculation:
    • Cosine theta × light intensity × material color = shaded color
  • Adjusts RGB values for lightness/darkness range
  • Minimum value handling:
    • Prevents pitch black surfaces (cosine theta = 0 or negative)
    • Adds ambient light intensity × surface color
    • Ambient light higher in daytime, closer to zero at night

Multiple Light Sources

  • Calculation performed multiple times with different light directions/intensities
  • Individual contributions added together
  • Computational intensity limits:
    • Scenes limit number of individual light sources
    • Sometimes limit range of influence
    • Distant lights ignored by triangles

Brilliant.org Sponsorship

  • Free 30-day trial with full access
  • Courses relevant to video game graphics:
    • Vector calculus, trigonometry, 3D geometry
    • Linear algebra, matrix math
    • Thinking and code for computational problem solving
  • 20% off annual subscription for first 200 viewers
  • Link: brilliant.org/BranchEducation

Flat vs Smooth Shading

  • Flat shading problem:
    • Each triangle has single normal
    • Same color throughout triangle surface
    • Unrealistic on curved surfaces
  • Smooth shading solution:
    • One normal per vertex (average of adjacent triangle normals)
    • Barycentric coordinates method
    • Creates smooth gradient of normals across triangle surface
    • Pixel-by-pixel coloring with smooth shading

Barycentric Coordinates Process

  • Takes center of each pixel
  • Uses vertex normals and coordinates of pre-rasterized triangle
  • Calculates barycentric normal for specific pixel
  • Proportional mix of three vertex normals
  • Results in gradient of normals and angles facing light
  • Smooth shading across curved surfaces

Production Details

  • 540 hours spent on research, writing, modeling RDR2 scene, and animation
  • Request for likes, subscriptions, shares, and comments

Advanced Topics Overview

  • Ray tracing applications:
    • Highly detailed scenes with accurate lighting/reflections
    • Single frame can take dozens of minutes to render
    • Used in certain games for shadows, reflections, improved lighting
  • DLSS (Deep Learning Super Sampling):
    • Upscales low resolution frame to 4K using convolutional neural network
    • Executed after ray tracing in graphics pipeline
  • Modern GPU architecture:
    • Three separate computational resources/cores:
      • CUDA/shading cores: execute graphics rendering pipeline
      • Ray tracing cores: self-explanatory function
      • Tensor cores: run DLSS
    • High-end games utilize all computational resources simultaneously
    • 4K games render frames in <10 milliseconds
    • CUDA/shading cores alone would take ~50 milliseconds per frame

Future Content Plans

  • Separate videos planned for ray tracing and DLSS topics
  • Considering additional video on advanced topics:
    • Shadows, reflections, UVs, normal maps
  • Request for viewer interest in comments

Channel Information

  • Branch Education creates 3D animations exploring modern technology
  • Supported by Patreon and YouTube membership sponsors
  • Links available in description for support

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome