AI summary by TwinMind
Stefano Puntoni | Decision-Driven Analytics | Talks at Google
Talks at Google
31m 18s
Book Introduction and Core Concept
- Dr. Stefano Puntoni (Wharton marketing professor, co-director AI at Wharton) discusses new book "Decision-Driven Analytics: Leveraging Human Intelligence to Unlock the Power of Data"
- Co-authored with Dr. Bart Dang, launching May 14
- Available on all major platforms
- Book addresses shift from data-driven to decision-driven analytics
Fundamental Paradigm Shift: Data-Driven vs Decision-Driven
- Traditional data-driven approach: invest in data infrastructure, build data lakes/oceans, then find uses
- Problem: puts emphasis on data as focal point rather than business objectives
- Decision-driven approach: start with business objectives and decisions, then determine needed data
- Data should serve decision-making, not drive it
- Process: identify alternatives → formulate questions → collect relevant data → analyze → decide
Two Types of People in Organizations
- Divers: excited by data puzzles, enjoy spreadsheets, crossups, plotting, running models
- Motivated by understanding that good evidence leads to good decisions
- Tend to be analysts, more geeky profiles
- Runners: focused on "what are we trying to do here," time-conscious, resource-limited
- Oriented toward making decisions and moving forward
- Tend to be managers/decision makers
- Gap between these profiles widening as data becomes more complex
- More complex data makes divers go deeper, runners become more impatient
Problems with Current Approach
- Managers delegating analytics entirely to analysts ("analytics not for me")
- Analysts need guidance on what organization is trying to achieve
- Telling analysts to "find something in there" rarely leads to success
- As AI systems become more complex, managers need MORE involvement, not less
- Massive investments often result in confusion and unproductive data use
Role of Decision Makers vs Analysts
- Decision makers' job: formulate very good questions (requires no data expertise)
- Key tasks for managers:
- Understand what could be undertaken to solve problems
- Create short list of best courses of action
- Figure out how to recognize which course is best
- Common problem: managers ask analysts vague questions
- Examples: "What should be our brand purpose?" "Are customers engaged with brand positioning?"
- These are impossible to answer with data or lack actionable outcomes
Three Types of Analytics Approaches
1. Preference-Driven Analytics (Worst)
- Boss tells analysts to "find data that tells them right"
- Seeking evidence to confirm preconceptions
- Creates veneer of objectivity for gut decisions
- Most dangerous because adds false confidence to biased decisions
2. Data-Driven Analytics (Traditional)
- Focus on data infrastructure and capabilities first
- Then try to find applications
- Often leads to solutions looking for problems
3. Decision-Driven Analytics (Recommended)
- Start with business objectives and decision alternatives
- Formulate specific questions data can answer
- Collect and analyze relevant data
- Make evidence-based decisions
Safeguards Against Bias
- Intellectual humility: acknowledge limitations of knowledge
- Open-mindedness: talk to people with different perspectives
- Diversity importance: prevents getting stuck thinking same way
- Most important reason for organizational diversity
- Diverse input leads to broader range of considered options
- Pre-registration: commit to analysis plan before seeing data
- Decide upfront what to test and how to judge success
- Specify statistical analysis methods in advance
- Prevents post-hoc rationalization and model shopping
Data Exploration vs Hypothesis Testing
- Data exploration (induction) is valuable and important
- Should be clearly distinguished from hypothesis testing (deduction)
- Exploration can generate insights and research questions
- But decision-making should use pre-committed analytical approaches
- Keep exploration separate from formal decision analysis
Precision vs Accuracy in Data Presentation
- Business presentations often show false precision (e.g., "78.2% of people")
- Point estimates alone insufficient to characterize any phenomenon
- Always need both position (mean) and dispersion (variance)
- Managers should always ask about dispersion around estimates
- Example: customer segments often have more variation within segments than between segments
- Failing to acknowledge uncertainty leads to oversimplification and bad decisions
Uncertainty and Forecasting
- Much business data involves:
- Proxies for unobservable phenomena
- Predictions about future events
- Always qualify estimates with ranges
- Sales forecasts should include potential variation (e.g., X/10 to X*10)
- Understanding uncertainty crucial for good decision-making
- Scatter plot example: 0.6 correlation looks strong but shows significant noise
AI and Human-Machine Complementarity
- AI has ceased being purely technical subject, now social science question
- Big question: how do people integrate AI to make good decisions?
- Focus should be on complementing human expertise, not replacing it
- Domain experts have years of valuable expertise
- Goal: develop analytics/AI systems that capitalize on human intelligence
- Partnership between domain experts, analysts, and AI systems
- Complementarity between human and machine will propel organizations forward
