AI summary by TwinMind
Tesla AI Day Highlights | Lex Fridman
Lex Fridman
11m 29s
Overall Assessment
- Tesla AI Day presented most amazing real-world AI and engineering effort speaker has ever seen
- Not primarily about Tesla bot, but autonomous driving and robotics perception/planning tasks
- Problems harder than people think, requiring massive scale effort in algorithms, data annotation, simulation, inference compute, training compute
- First time seeing kind and scale of effort with chance to solve autonomous driving and general robotics problems
Neural Network Architecture Innovations
- Vector Space Prediction: Predict in vector space not image space
- Big leap beyond usual computer vision operating in 2D image space
- Reality happens in 3D world, doesn't make sense doing ML on 2D projections
- Obvious but very difficult engineering step
- Camera Sensor Fusion: Fusion before detections
- Multi-scale feature level fusion currently
- Detection/ML on all sensors combined vs individually then combining decisions
- Video Context Modeling: Using video to model vector space and time
- Each frame concatenates positional encodings, multi-cam features, ego kinematics
- Spatial recurrent neural network architecture forming 2D grid around car
- Each grid cell is RNN
- Can build map in RNN feature space, do planning in that space
- Future Improvements: Moving fusion earlier in network
- Currently space/time fusion late in network
- Earlier fusion moves toward full end-to-end driving with multiple modalities
- Neural Network Planning: Least utilization currently
- Optimal planning in action space intractable, need heuristics
- Use neural networks as heuristics similar to Monte Carlo tree search in MuZero/AlphaZero
- Significantly prunes search through action space, avoids local optima
Data and Annotation Systems
- Manual Labeling:
- Label in vector space for neural networks predicting in vector space
- Tesla hired in-house team of annotators with custom tools
- Perform labeling in vector space, project to image space
- Saves work, direct annotation in prediction space
- Auto Labeling: Key to self-supervised learning
- Uses clips with video, IMU, GPS, odometry from multiple vehicles at same location/time
- Generates labels for static world and moving objects with kinematics
- Vehicles annotate each other through registration
- Stronger as fleet grows, catches more edge cases
- Simulation: For rare edge cases and complex scenes
- Simulate cases not appearing often even in large datasets
- Annotation of ultra-complex scenes (example: 100 pedestrians)
- Where accurate real-world labeling impossible
Autopilot Computer Infrastructure
- Neural network compiler optimizing latency
- Testing/debugging tools for candidate neural networks
- Developer tools for deployed neural networks
- Almost 10,000 GPUs currently used for continual retraining
- Network fully retrained end-to-end every week or two weeks
Dojo Computer System
- Architecture: Single training tile = 9 ExaFLOPS
- Made of D1 chips built in-house by Tesla
- Each chip with super fast I/O
- Each tile with super fast I/O for arbitrary connection
- Power supply and cooling integrated
- Scale: Connected ~1 million nodes
- 1.1 ExaFLOP compute center
- Arbitrarily scalable
- Contending to be world's most powerful neural network training computer
Iterative Data Engine
- Process Loop: Auto labeling + manual labeling of edge cases → data collection → retraining → deploying → back to data collection
- Can iterate arbitrarily to improve network performance
- No ceiling to this improvement loop
- Tesla AI machine can improve arbitrarily through this process
Broader Applications and Future
- AI Training as Service: Dojo can compete with AWS/Google Cloud
- Not limited to autopilot computer
- Simple PyTorch deployment across nodes
- Usable for any machine learning problem requiring scale
- Beyond Driving: Neural network architecture and data engine applicable to:
- Home environments
- Factory settings
- Any robot form with cameras and actuators
- Including humanoid form
Tesla Bot and Robotics Vision
- Humanoid Tesla bot presentation exciting for robotics enthusiasts
- Two parallel problems to solve:
- Perception, movement, object manipulation (Tesla bot may solve)
- Human-robot interaction and friendship (speaker hopes to contribute)
- Tesla bot could solve boring/dangerous tasks problem
- Paints picture of exciting future
Technical Challenges and Unknowns
- Autonomous driving problem difficulty still unknown
- Need for driver sensing remains important
- Human-robot interaction problem needs solving for pleasant experience
- Manual and auto labeling loop leading to retraining and deployment is incredible innovation
