AI summary by TwinMind
Eric Siegel | The AI Playbook | Talks at Google
Talks at Google
59m 18s
Introduction and Speaker Background
- Eric Siegel presenting at Toxic Google, introduced by Brett Durett
- 30+ years in machine learning, founder/CEO of Gödel AI
- Founder of Machine Learning Week conference (running since 2009), upcoming in Phoenix in 2 weeks
- Author of "Predictive Analytics" bestseller and new book "The AI Playbook"
- Forbes contributor, former Columbia University professor
Core Problem: ML Deployment Failure
- Most new machine learning projects fail to achieve deployment
- Root cause: conceived as technology projects rather than business endeavors/operations improvement projects
- "Consulting gig not a technology install" - requires business consulting approach
- Both data scientists and business stakeholders avoid responsibility
- Data scientists: "that's management's job"
- Business professionals: "I delegate technical stuff to data scientists"
- Need for established framework/paradigm/playbook for organizational ML projects
Predictive Analytics Value Proposition
- Business needs prediction → prediction requires ML → ML requires data
- Data → ML → models that predict → predictions drive large-scale operations
- Applications: cut costs, boost sales, streamline manufacturing, combat risk, prevent fraud, fortify healthcare, conquer spam, win elections
- Core mechanism: order/prioritize decisions from highest opportunity/lowest risk
- Universal pattern: ranking and drawing decision thresholds
The Prediction Effect and Credibility
- "Little prediction goes a long way" - predicting better than guessing generally sufficient
- Example calculation: Mass marketing vs targeted marketing
- Mass marketing: $2M cost, 1% response rate, $2.2M revenue, $0.2M profit
- Targeted marketing (top 25%, 3x lift): 75% cost savings, $1.15M profit (5x improvement)
- Data discoveries examples:
- Credit card at drinking establishment = higher credit risk
- Spending at dentist = lower credit risk
- Buying felt pads = lower credit risk
- Liking curly fries on Facebook = more intelligent
- Skipping breakfast (young males) = higher coronary heart disease risk
Technical Implementation
- Decision trees as interpretable models example
- Chase bank mortgage defection prediction rule: mortgage $67-183K, interest rate >8.7%, loan-to-value <87% = 25.6% defection chance
- Ensemble models: population of trees voting together
- Neural networks become "soup of math" - less transparent but potentially better performance
- Transformers (invented at Google) basis for generative AI
Real-World Deployment Example: UPS
- Package Flow Technology + Orion system
- Predicts tomorrow's deliveries for optimization across 1,000 US shipping centers
- Augments known packages with predicted ones for complete planning picture
- Results: 185 million miles/year savings, $350M savings, 8M gallons fuel saved, 185K metric tons emissions reduced
- Required significant battles for approval and deployment
Three Critical Understanding Elements
- What's predicted (dependent variable)
- How well (metrics)
- What's done about it (deployment)
- Semi-technical understanding needed by stakeholders - "much more accessible than high school algebra"
Metrics Problem
- Data scientists focus on technical metrics (precision, recall, AUC, accuracy)
- Need business metrics: profit, revenue, savings, returns
- Technical metrics show abstract predictive performance vs baseline
- Business metrics show absolute value depending on deployment
- Translation between technical and business metrics rarely seen
- Profit curves almost never generated despite being accessible
Six-Step BizML Playbook
- Deployment Goal: Define what's predicted and what's done about it (reverse planning)
- Technical Objective: Detailed definition of dependent variable with business stakeholder involvement
- Example: "buy within 13 days, purchase ≥$125 after shipping, no return within 45 days"
- Metrics: Both technical and business metrics, thresholds for deployment
- Data Preparation: Get data in right format (biggest technical bottleneck)
- Model Training: The machine learning/rocket science part
- Deployment: Where actual value is captured - most often fails
Deep Collaboration Requirement
- End-to-end collaboration across project lifecycle
- Business stakeholders must ramp up on semi-technical understanding
- "If stakeholder doesn't get hands dirty, their feet will get cold"
- Projects are business endeavors using ML, not pure technology installs
AI Hype vs Reality
- "AI first means using AI last" - fall in love with problem, not solution
- AI's "original sin" was calling it AI - anthropomorphism problem
- Intelligence not a metric - subjective concept, goal posts keep moving
- Generative AI overhyped by factor of 10-20, but still valuable at 10% of hype
- No concrete step toward AGI despite progress
- Neural networks scale well at emulating human behavior from limited observations
Generative AI Deployment Examples
- Ali Financial Bank: 34% time savings in marketing campaign creation
- Fortune 500 software company customer service: 14% improvement in issues resolved per hour
- 34% improvement for novice/low-skilled workers
- Requires human proofreading - less autonomous than predictive AI
Current Opportunities and Challenges
- Top use cases: targeting marketing, churn modeling, credit scoring, fraud detection
- Long tail of applications across industries
- Deployment failure rate high - often swept under rug due to AI hype
- Need to organize businesses around predictive use cases from get-go rather than afterthought
- Data preparation biggest technical challenge - time leak issues, rolling back time problem
Future Direction
- Educational need for non-data scientists on semi-technical understanding
- Data scientists need broader perspective beyond number crunching
- No established best practices yet for LLM/generative AI enterprise projects
- BizML framework specialized for predictive, but broad concepts apply to generative AI
