AI summary by TwinMind
Jeff Dean: AI isn't as smart as you think -- but it could be | TED
TED
18m 31s
Jeff Dean Background & AI Progress Overview
- Jeff Dean, AI Research & Health lead at Google, 20+ years at company
- Started in tiny office above T-Mobile store in downtown Palo Alto
- Last decade: tremendous AI progress in computer vision, language understanding, speech
- Computer vision transformation: 10 years ago computers couldn't see, now they can
- Applications: flood prediction, 100+ language translation, disease prediction/diagnosis
Neural Networks & Computational Power Foundation
- Two key components underlying AI progress:
- Neural networks: breakthrough approach, shining last 15 years (not new idea)
- Computational power: required to make neural networks effective
- Neural networks loosely based on real neural systems
- Ideas from 1960s-70s
- Individual neuron: set of inputs with associated weights, output = function of inputs × weights
- Learning: repeatedly making tiny adjustments to weight values, strengthening influences
- Can train for complex tasks: translation, object detection
Historical Context & Early Challenges
- Since computing beginning: people wanted computers to see, understand language/speech
- Early approaches: hand-coded algorithms, didn't work well
- Jeff's 1990 undergraduate experience:
- Took neural networks class, excited about potential
- University of Minnesota had 32-processor machine
- Senior thesis: parallel training of neural networks
- Was wrong about computational needs
- Required ~1 million times more computational power than available in 1990
- 2005: Moore's law provided necessary computing power
- Universities worldwide started seeing neural network successes
Google's Neural Network Breakthrough
- Google team started project to train very large neural networks
- Trained system with 10 million randomly selected YouTube video frames
- System developed object recognition capabilities
- Notably: learned to recognize cats without being told what cats were
- Used patterns in data to develop cat concept independently
- Beginning of decade-long string of successes at Google and elsewhere
- Applications: speech recognition, search quality, maps geographic understanding
TPU Development
- Built hardware tailored for neural network computations
- Neural networks have two special properties:
- Tolerant of reduced precision (couple significant digits vs 6-7)
- Algorithms composed of matrix and vector operations sequences
- TPU v1 (Tensor Processing Unit): specialized for low-precision matrix/vector operations
- Used behind every Google search, translation, DeepMind AlphaGo matches
- Lee Sedol and Ke Jie competed against racks of TPU cards
- Built subsequent improved TPU versions
Three Key Problems with Current AI
Problem 1: Single-Task Models
- Most neural networks trained for one thing only
- Heavyweight process: curate dataset, decide architecture, initialize weights, apply computation
- Results in thousands of separate models for different tasks
- Unlike human learning: people leverage existing knowledge for new skills
- Example: Jeff's hydroponic gardening - didn't need to relearn basic plant knowledge
- Current approach like "forgetting entire education every time you try something new"
Problem 2: Single Modality Focus
- Most models deal with only one data type: images OR text OR speech
- Humans continuously use all senses to learn, react, take actions
- Should build models that process text, images, speech simultaneously
- Fuse modalities so "leopard" concept triggered whether seeing word, video, or hearing spoken word
- Can extend to non-human inputs: genetic sequences, 3D point clouds
Problem 3: Dense Models
- Single model fully activated for every task/example
- Same activation whether simple or complicated task
- Unlike human brains: different parts good at different things
- Example: watching garbage truck back up - Shakespearean sonnet brain part inactive
- Should use sparsely activated models: call upon relevant parts for specific tasks
- Advantage: high-capacity but efficient, only using needed parts
Pathways System Solution
- Building system enabling new approaches called "Pathways"
- Capabilities:
- Handle thousands/millions of different tasks
- Incrementally add new tasks
- Deal with all modalities simultaneously
- Learn new tasks as needed
- Call upon relevant model bits for different examples/tasks
- Expected to be step forward in AI system building
Responsible AI Considerations
- Need to ensure powerful AI systems benefit everyone
- Important questions: fairness, interpretability, privacy, security
- Training on thousands/millions of tasks requires large data amounts
- Data must be thoughtfully collected, representative of different communities/situations worldwide
- 2018: Google published AI principles for technology development
- Guides research approach and product AI usage
- Principles updated as learning continues
- Hundreds of people working on related research issues
Future Applications & Impact
- Move from single-purpose pattern recognition to general-purpose intelligent systems
- Deeper world understanding enables tackling greatest humanity problems:
- Diagnose more diseases
- Engineer better medicines (chemistry/physics knowledge infusion)
- Advance educational systems (individualized tutoring)
- Tackle climate change, clean energy solutions
- Requires multidisciplinary expertise worldwide
- AI potential to help billions of people vs current millions
Q&A Session with Chris Anderson
AI Pattern Recognition Evolution
- Traditional AI: computers recognize information patterns, get better than humans
- New approach: richer-layered concepts including various components (leopard example)
- Grand challenge: generalize from known tasks to new tasks easily
- Current separate models need lots of data, learning everything from nothing
- Pathways approach: systems infused with thousands/millions of tasks knowledge
- Can teach new tasks with relatively few examples (5 vs 10,000-100,000)
Potential Risks & Safeguards
- AI can be powerful force for good or negative consequences depending on application
- Importance of principles for evaluating AI uses, careful/thoughtful application consideration
- Concern: AI learning from world "as it is" vs "as it should be"
- Research focus: adapt/augment data to reflect desired values vs observed world values
- Hundreds of people working on fairness, interpretability research topics
Google's Research Independence
- Research group collaborates with Ads, Search, Maps groups
- Also conducts basic research published openly
- 1,000+ papers published last year on various topics including fairness, interpretability
- Need to advance state of art for safe, responsible model development
- Importance of showing world this work creates better future for Google and everyone
