AI summary by TwinMind
The Hard Tradeoffs of Edge AI Hardware
Asianometry
14m 11s
Introduction & Context
- Recent deep learning AI models (GitHub Copilot, GPT-3, DALL-E 2) approaching uncanny performance levels
- Core challenge: deploying powerful models where needed most - in field with customers
- Focus: challenges of running AI on edge devices vs cloud-based solutions
Edge Device Definition & Constraints
- Edge devices: connected to internet but closer to users than data centers
- Examples: robotics, unmanned aircraft, remote sensing satellites, home entertainment devices (Amazon Echo), wearables, smartphones
- Key constraints: power management, processing speed, memory limitations
- Self-driving car AI chips prioritize latency (speed to result)
- Small commercial drones: 20-30 minutes flight time, <5% power budget for computing/data processing
Typical Edge AI Applications
- Computer vision and audio processing jobs predominant
- Specific tasks: natural language processing, facial recognition, traffic prediction
Model Size Growth Problem
- 2012: AlexNet state-of-the-art - 61 million weights
- 2021: CoCa ImageNet winner - 2.1 billion weights (34x increase)
- Neural networks consume 13,500x more energy than histogram of oriented gradients for object detection
- Memory access most energy-intensive operation: 200x more energy than multiply-accumulate operation
Cloud vs Edge Approaches
Cloud Offloading (Thin Client)
- Early approach: Siri, Amazon Echo relay information to servers
- Issues: data transmission latency, connection stability, privacy concerns
Hybrid Approach
- Edge hardware first pass, cloud final confirmation
- Shares downsides of both approaches
- Must maintain models in both environments
Neural Network Model Optimization Techniques
Compact Model Training
- SqueezeNet and MobileNet examples
- Replace traditional structures to reduce weights
- SqueezeNet claims: AlexNet-level accuracy with 50x fewer weights, compressed to 0.5MB
Post-Processing Optimization
Weight Quantization
- Change weight storage format: 32-bit floating point to 8-bit fixed point
- Results in >12% accuracy loss
- Reduces memory footprint through matrix shrinkage
Pruning
- Remove redundant weights
- One paper claims 95% of neural network weights highly correlated to few key weights
- Can retain much accuracy while removing redundant parameters
Trade-off Reality
- No free lunch: accuracy vs memory/power usage trade-offs
- Optimization methods consistently miss expectations
- Difficult to predict optimization effects on performance and resource usage
Edge AI Hardware Options
CPUs/Microcontrollers (MCUs)
- Examples: Raspberry Pi, tiny MCUs with 100KB RAM
- Advantages: easy programming, versatile, low power, cheap
- Disadvantages: not parallel (even multi-core versions)
- 250 billion microcontrollers deployed in field
- TensorFlow Lite for microcontrollers enables projects like voice-enabled bike controllers
- TinyML field has significant impact potential
GPUs
- Originally designed for gaming, massively parallel
- Easily programmable via NVIDIA CUDA
- Great for training AI models
- Disadvantage: energy hungry, less suitable for edge inference
- Example: NVIDIA Jetson Nano - $99 embedded computer, Raspberry Pi-like
Field Programmable Gate Arrays (FPGAs)
- Programmable logic blocks and routing interconnections
- Inherently parallel like GPUs
- Programmed using VHDL and Verilog hardware description languages
- Advantages:
- Flexible, reconfigurable as needed
- Useful for autonomous cars (changing rules/algorithms)
- Energy efficient due to block RAM (embedded memory blocks)
- Significantly more power efficient for models fitting in block RAM
- Disadvantages:
- Less memory bandwidth and computing resources than GPUs
- Some devices: as little as 10MB on-chip memory
- Requires design expertise (VHDL/Verilog vs C/C++)
Application-Specific Integrated Circuits (ASICs)
- Custom processors for specific tasks
- AI accelerators are ASIC class
- Advantages: optimized for specific use case
- Disadvantages:
- Substantial upfront financial and human resources
- Leading edge process nodes cost millions
- Cannot change architectures after fabrication
- Most companies build for generic functionality
Commercial Edge AI Products
Tech Giants
Intel
- Movidius Myriad X VPU (Vision Processing Unit) - released 2020
- Applications: drones, robotics, smart cameras
- Movidius: Irish startup acquired by Intel 2016
- Edge TPU purpose-built for edge inference
- Sold through Coral initiative
- Available as USB sticks and developer boards
NVIDIA
- Tegra series system-on-chips
- Mobile but not strictly edge AI accelerator
Smaller Companies
Rockchip
- Chinese fabless semiconductor maker, Fujian Province
- RK1808 standalone Neural Processing Unit (NPU)
- Also sold as USB package "Toy Brick" for easier integration
Kneron (Milpitas, California)
- Small, low-power, low-cost chips
- Neural accelerators pair with other processors
- Handle image recognition and object detection
Neuhon (Hsinchu)
- Operating since 2015
- AI chips for speech and body motion recognition
Hardware-Software Co-Design Challenges
- Hardware and software extremely tightly bound
- Tweaking one affects the other, slowing iteration and progress
- Need faster development cycles
Hardware-Aware Neural Architecture Search
- Include hardware variables in neural network model
- Optimize for specific hardware (GPU or FPGA)
- Doesn't work well with ASICs due to customization flexibility
Hardware-Algorithm Co-Design
- ASIC advantage: simultaneous co-design of hardware and algorithms
- Similar to design-technology co-optimization in chip manufacturing
- Significant potential for edge AI hardware space
Future Outlook
- Massive models more powerful than ever
- Edge AI hardware faces challenging economic and physical limits
- Historical parallel: computers unlocked unprecedented benefits in 20th century industry/commerce
- AI has similar potential but requires satisfactory edge hardware
- Risk: AI's full potential remains locked in ephemeral cloud without edge advancement
- Industry must continue evolving and pushing forward
