AI summary by TwinMind

What is AutoML? A conversation with Gnosis Data Analysis

StatQuest with Josh Starmer

15m 20s

Introduction and Sponsorship

  • StatQuest sponsored by JADBio (Just Add Data)
  • Automatic machine learning platform - "just add data and algorithms do the rest"
  • Interview with Giannis, professor at University of Crete and CEO of Gnosis Data Analysis (JADBio product)

What is AutoML

  • AutoML = Automated Machine Learning
  • Goal: fully automate end-to-end machine learning process
  • Practice: throw data in, click button, get results
  • "Declare what you want, system knows how to compute it"
  • Term should be taken with grain of salt - gradual automation of bigger portions
  • No systems can fully automate ALL types of ML yet
    • JADBio: fully automates predictive modeling with tabular data
    • Does not handle: unsupervised learning, sequences, graphs, causal discovery

AutoML vs CASH Systems

  • People confuse AutoML with CASH (Combined Algorithm Selection and Hyper-parameter optimization)
  • CASH algorithms optimize:
    • Algorithm combinations at each analysis step
    • Transformations, imputation, feature selection, modeling
    • Hyper-parameters
  • CASH only returns predictive model (example: Auto-sklearn)
  • True AutoML should provide much more:
    • Performance estimates
    • Feature selection
    • Model interpretation/explanation
    • Decision support
    • Operational deployment
    • Monitoring and alarms
    • "Everything a good analyst would provide"

Target Audience

  • AutoML should be for anyone - not just coders
  • From CEOs analyzing company data to home users with Excel sheets
  • Should not require: coding, math, statistics, or ML knowledge
  • Democratic approach - "bring ML to the masses"
  • Vision: follow Excel's path (initially for accountants, now universal)
  • Also boosts expert productivity significantly
  • Experts appreciate AutoML because they understand the difficulties
  • Analogy: programmers moved from assembly to Python (higher-level automation)
  • Experts can focus on: data representation, results interpretation, presentation

Key Considerations for AutoML Users

  • Critical issue: correctness of performance estimation
  • Multiple comparisons problem in induction algorithms
  • Also called "winner's curse" in statistics
  • Problem: trying 1000 analysis pipelines leads to overestimation
    • Best model with 90% cross-validated accuracy is overestimated on average
    • Cross-validating one algorithm = right performance estimate
    • Cross-validating 1000 algorithms + choosing best = overestimated performance
  • Worse for: small sample sizes, very imbalanced data
  • Holdout sets lose samples for training (unacceptable with limited data)
  • Need statistical procedures to remove optimism (like multiple testing corrections)
  • JADBio has methods to compensate for this effect

AutoML vs Manual Optimization

  • Common attitude: "I can do better than automated systems"
  • Difficult to prove scientifically due to:
    • Which expert to compare against?
    • Different expertise levels
    • AutoML constantly improving
  • Case study: FDA brain cancer survival prediction challenge
    • JADBio: downloaded data, uploaded, clicked, got results in minutes
    • Result: 3rd place out of 30 groups
    • Winner: only 1 point higher C-index (within confidence intervals)
    • Winner likely spent hundreds of hours vs minutes
    • JADBio estimated performance within 1 decimal point accuracy
    • Winner estimated 100% performance, actually got 74% (0.74 C-index)
  • Human experts still needed for:
    • Data preprocessing and cleaning
    • Data representation in suitable format
    • Problem formulation as ML task
    • Making data ready for AutoML

Impact on Data Science Jobs

  • Historical precedent: assembly language to Python didn't eliminate programmers
  • Global economy needs more programmers despite automation
  • Currently only 5% of global data is analyzed
  • AutoML will increase analysis demand
  • Businesses/scientists will unlock data value and want more
  • AutoML boosts productivity but won't eliminate expert need (foreseeable future)
  • Experts still needed for:
    • Problem representation and formulation
    • Results interpretation
    • Correct model application
  • Future experts must be more specialized/educated/sophisticated
  • Simple tasks (cross-validation, random forest) will be automated with higher quality
  • Next generation data scientists must add more value to justify salaries

Future of AutoML

  • "Very bright" future predicted
  • Era of manual ML scripting reaching critical point
  • Prediction: within 5 years, most data analysis will be completely automated
  • Or at minimum: AutoML as major part of analysis
  • Expert scripting will evolve to:
    • Calling AutoML systems
    • Configuring AutoML systems
    • Customizing AutoML systems
    • Higher-level scripting
  • Great business opportunities for startups and investments
  • Specialized AutoML systems emerging (JADBio caters to bioinformatics)
  • Initial proliferation then consolidation to few survivors
  • Companies will need embedded AutoML to survive:
    • Database vendors
    • Data ranking companies
    • Cloud providers
    • Convert stored data to actionable knowledge automatically
  • AutoML embedded in devices (laptops, cars) for behavioral analysis
  • Systems will analyze user behavior/usage/environment interaction to become smarter

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome