AI summary by TwinMind

Eric Siegel | The AI Playbook | Talks at Google

Talks at Google

59m 18s

Introduction and Speaker Background

  • Eric Siegel presenting at Toxic Google, introduced by Brett Durett
  • 30+ years in machine learning, founder/CEO of Gödel AI
  • Founder of Machine Learning Week conference (running since 2009), upcoming in Phoenix in 2 weeks
  • Author of "Predictive Analytics" bestseller and new book "The AI Playbook"
  • Forbes contributor, former Columbia University professor

Core Problem: ML Deployment Failure

  • Most new machine learning projects fail to achieve deployment
  • Root cause: conceived as technology projects rather than business endeavors/operations improvement projects
  • "Consulting gig not a technology install" - requires business consulting approach
  • Both data scientists and business stakeholders avoid responsibility
    • Data scientists: "that's management's job"
    • Business professionals: "I delegate technical stuff to data scientists"
  • Need for established framework/paradigm/playbook for organizational ML projects

Predictive Analytics Value Proposition

  • Business needs prediction → prediction requires ML → ML requires data
  • Data → ML → models that predict → predictions drive large-scale operations
  • Applications: cut costs, boost sales, streamline manufacturing, combat risk, prevent fraud, fortify healthcare, conquer spam, win elections
  • Core mechanism: order/prioritize decisions from highest opportunity/lowest risk
  • Universal pattern: ranking and drawing decision thresholds

The Prediction Effect and Credibility

  • "Little prediction goes a long way" - predicting better than guessing generally sufficient
  • Example calculation: Mass marketing vs targeted marketing
    • Mass marketing: $2M cost, 1% response rate, $2.2M revenue, $0.2M profit
    • Targeted marketing (top 25%, 3x lift): 75% cost savings, $1.15M profit (5x improvement)
  • Data discoveries examples:
    • Credit card at drinking establishment = higher credit risk
    • Spending at dentist = lower credit risk
    • Buying felt pads = lower credit risk
    • Liking curly fries on Facebook = more intelligent
    • Skipping breakfast (young males) = higher coronary heart disease risk

Technical Implementation

  • Decision trees as interpretable models example
  • Chase bank mortgage defection prediction rule: mortgage $67-183K, interest rate >8.7%, loan-to-value <87% = 25.6% defection chance
  • Ensemble models: population of trees voting together
  • Neural networks become "soup of math" - less transparent but potentially better performance
  • Transformers (invented at Google) basis for generative AI

Real-World Deployment Example: UPS

  • Package Flow Technology + Orion system
  • Predicts tomorrow's deliveries for optimization across 1,000 US shipping centers
  • Augments known packages with predicted ones for complete planning picture
  • Results: 185 million miles/year savings, $350M savings, 8M gallons fuel saved, 185K metric tons emissions reduced
  • Required significant battles for approval and deployment

Three Critical Understanding Elements

  • What's predicted (dependent variable)
  • How well (metrics)
  • What's done about it (deployment)
  • Semi-technical understanding needed by stakeholders - "much more accessible than high school algebra"

Metrics Problem

  • Data scientists focus on technical metrics (precision, recall, AUC, accuracy)
  • Need business metrics: profit, revenue, savings, returns
  • Technical metrics show abstract predictive performance vs baseline
  • Business metrics show absolute value depending on deployment
  • Translation between technical and business metrics rarely seen
  • Profit curves almost never generated despite being accessible

Six-Step BizML Playbook

  1. Deployment Goal: Define what's predicted and what's done about it (reverse planning)
  2. Technical Objective: Detailed definition of dependent variable with business stakeholder involvement
    • Example: "buy within 13 days, purchase ≥$125 after shipping, no return within 45 days"
  3. Metrics: Both technical and business metrics, thresholds for deployment
  4. Data Preparation: Get data in right format (biggest technical bottleneck)
  5. Model Training: The machine learning/rocket science part
  6. Deployment: Where actual value is captured - most often fails

Deep Collaboration Requirement

  • End-to-end collaboration across project lifecycle
  • Business stakeholders must ramp up on semi-technical understanding
  • "If stakeholder doesn't get hands dirty, their feet will get cold"
  • Projects are business endeavors using ML, not pure technology installs

AI Hype vs Reality

  • "AI first means using AI last" - fall in love with problem, not solution
  • AI's "original sin" was calling it AI - anthropomorphism problem
  • Intelligence not a metric - subjective concept, goal posts keep moving
  • Generative AI overhyped by factor of 10-20, but still valuable at 10% of hype
  • No concrete step toward AGI despite progress
  • Neural networks scale well at emulating human behavior from limited observations

Generative AI Deployment Examples

  • Ali Financial Bank: 34% time savings in marketing campaign creation
  • Fortune 500 software company customer service: 14% improvement in issues resolved per hour
  • 34% improvement for novice/low-skilled workers
  • Requires human proofreading - less autonomous than predictive AI

Current Opportunities and Challenges

  • Top use cases: targeting marketing, churn modeling, credit scoring, fraud detection
  • Long tail of applications across industries
  • Deployment failure rate high - often swept under rug due to AI hype
  • Need to organize businesses around predictive use cases from get-go rather than afterthought
  • Data preparation biggest technical challenge - time leak issues, rolling back time problem

Future Direction

  • Educational need for non-data scientists on semi-technical understanding
  • Data scientists need broader perspective beyond number crunching
  • No established best practices yet for LLM/generative AI enterprise projects
  • BizML framework specialized for predictive, but broad concepts apply to generative AI

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome