AI summary by TwinMind

Stefano Puntoni | Decision-Driven Analytics | Talks at Google

Talks at Google

31m 18s

Book Introduction and Core Concept

  • Dr. Stefano Puntoni (Wharton marketing professor, co-director AI at Wharton) discusses new book "Decision-Driven Analytics: Leveraging Human Intelligence to Unlock the Power of Data"
  • Co-authored with Dr. Bart Dang, launching May 14
  • Available on all major platforms
  • Book addresses shift from data-driven to decision-driven analytics

Fundamental Paradigm Shift: Data-Driven vs Decision-Driven

  • Traditional data-driven approach: invest in data infrastructure, build data lakes/oceans, then find uses
  • Problem: puts emphasis on data as focal point rather than business objectives
  • Decision-driven approach: start with business objectives and decisions, then determine needed data
  • Data should serve decision-making, not drive it
  • Process: identify alternatives → formulate questions → collect relevant data → analyze → decide

Two Types of People in Organizations

  • Divers: excited by data puzzles, enjoy spreadsheets, crossups, plotting, running models
    • Motivated by understanding that good evidence leads to good decisions
    • Tend to be analysts, more geeky profiles
  • Runners: focused on "what are we trying to do here," time-conscious, resource-limited
    • Oriented toward making decisions and moving forward
    • Tend to be managers/decision makers
  • Gap between these profiles widening as data becomes more complex
  • More complex data makes divers go deeper, runners become more impatient

Problems with Current Approach

  • Managers delegating analytics entirely to analysts ("analytics not for me")
  • Analysts need guidance on what organization is trying to achieve
  • Telling analysts to "find something in there" rarely leads to success
  • As AI systems become more complex, managers need MORE involvement, not less
  • Massive investments often result in confusion and unproductive data use

Role of Decision Makers vs Analysts

  • Decision makers' job: formulate very good questions (requires no data expertise)
  • Key tasks for managers:
    • Understand what could be undertaken to solve problems
    • Create short list of best courses of action
    • Figure out how to recognize which course is best
  • Common problem: managers ask analysts vague questions
    • Examples: "What should be our brand purpose?" "Are customers engaged with brand positioning?"
    • These are impossible to answer with data or lack actionable outcomes

Three Types of Analytics Approaches

1. Preference-Driven Analytics (Worst)

  • Boss tells analysts to "find data that tells them right"
  • Seeking evidence to confirm preconceptions
  • Creates veneer of objectivity for gut decisions
  • Most dangerous because adds false confidence to biased decisions

2. Data-Driven Analytics (Traditional)

  • Focus on data infrastructure and capabilities first
  • Then try to find applications
  • Often leads to solutions looking for problems

3. Decision-Driven Analytics (Recommended)

  • Start with business objectives and decision alternatives
  • Formulate specific questions data can answer
  • Collect and analyze relevant data
  • Make evidence-based decisions

Safeguards Against Bias

  • Intellectual humility: acknowledge limitations of knowledge
  • Open-mindedness: talk to people with different perspectives
  • Diversity importance: prevents getting stuck thinking same way
    • Most important reason for organizational diversity
    • Diverse input leads to broader range of considered options
  • Pre-registration: commit to analysis plan before seeing data
    • Decide upfront what to test and how to judge success
    • Specify statistical analysis methods in advance
    • Prevents post-hoc rationalization and model shopping

Data Exploration vs Hypothesis Testing

  • Data exploration (induction) is valuable and important
  • Should be clearly distinguished from hypothesis testing (deduction)
  • Exploration can generate insights and research questions
  • But decision-making should use pre-committed analytical approaches
  • Keep exploration separate from formal decision analysis

Precision vs Accuracy in Data Presentation

  • Business presentations often show false precision (e.g., "78.2% of people")
  • Point estimates alone insufficient to characterize any phenomenon
  • Always need both position (mean) and dispersion (variance)
  • Managers should always ask about dispersion around estimates
  • Example: customer segments often have more variation within segments than between segments
  • Failing to acknowledge uncertainty leads to oversimplification and bad decisions

Uncertainty and Forecasting

  • Much business data involves:
    • Proxies for unobservable phenomena
    • Predictions about future events
  • Always qualify estimates with ranges
  • Sales forecasts should include potential variation (e.g., X/10 to X*10)
  • Understanding uncertainty crucial for good decision-making
  • Scatter plot example: 0.6 correlation looks strong but shows significant noise

AI and Human-Machine Complementarity

  • AI has ceased being purely technical subject, now social science question
  • Big question: how do people integrate AI to make good decisions?
  • Focus should be on complementing human expertise, not replacing it
  • Domain experts have years of valuable expertise
  • Goal: develop analytics/AI systems that capitalize on human intelligence
  • Partnership between domain experts, analysts, and AI systems
  • Complementarity between human and machine will propel organizations forward

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome