AI summary by TwinMind
What is AutoML? A conversation with Gnosis Data Analysis
StatQuest with Josh Starmer
15m 20s
Introduction and Sponsorship
- StatQuest sponsored by JADBio (Just Add Data)
- Automatic machine learning platform - "just add data and algorithms do the rest"
- Interview with Giannis, professor at University of Crete and CEO of Gnosis Data Analysis (JADBio product)
What is AutoML
- AutoML = Automated Machine Learning
- Goal: fully automate end-to-end machine learning process
- Practice: throw data in, click button, get results
- "Declare what you want, system knows how to compute it"
- Term should be taken with grain of salt - gradual automation of bigger portions
- No systems can fully automate ALL types of ML yet
- JADBio: fully automates predictive modeling with tabular data
- Does not handle: unsupervised learning, sequences, graphs, causal discovery
AutoML vs CASH Systems
- People confuse AutoML with CASH (Combined Algorithm Selection and Hyper-parameter optimization)
- CASH algorithms optimize:
- Algorithm combinations at each analysis step
- Transformations, imputation, feature selection, modeling
- Hyper-parameters
- CASH only returns predictive model (example: Auto-sklearn)
- True AutoML should provide much more:
- Performance estimates
- Feature selection
- Model interpretation/explanation
- Decision support
- Operational deployment
- Monitoring and alarms
- "Everything a good analyst would provide"
Target Audience
- AutoML should be for anyone - not just coders
- From CEOs analyzing company data to home users with Excel sheets
- Should not require: coding, math, statistics, or ML knowledge
- Democratic approach - "bring ML to the masses"
- Vision: follow Excel's path (initially for accountants, now universal)
- Also boosts expert productivity significantly
- Experts appreciate AutoML because they understand the difficulties
- Analogy: programmers moved from assembly to Python (higher-level automation)
- Experts can focus on: data representation, results interpretation, presentation
Key Considerations for AutoML Users
- Critical issue: correctness of performance estimation
- Multiple comparisons problem in induction algorithms
- Also called "winner's curse" in statistics
- Problem: trying 1000 analysis pipelines leads to overestimation
- Best model with 90% cross-validated accuracy is overestimated on average
- Cross-validating one algorithm = right performance estimate
- Cross-validating 1000 algorithms + choosing best = overestimated performance
- Worse for: small sample sizes, very imbalanced data
- Holdout sets lose samples for training (unacceptable with limited data)
- Need statistical procedures to remove optimism (like multiple testing corrections)
- JADBio has methods to compensate for this effect
AutoML vs Manual Optimization
- Common attitude: "I can do better than automated systems"
- Difficult to prove scientifically due to:
- Which expert to compare against?
- Different expertise levels
- AutoML constantly improving
- Case study: FDA brain cancer survival prediction challenge
- JADBio: downloaded data, uploaded, clicked, got results in minutes
- Result: 3rd place out of 30 groups
- Winner: only 1 point higher C-index (within confidence intervals)
- Winner likely spent hundreds of hours vs minutes
- JADBio estimated performance within 1 decimal point accuracy
- Winner estimated 100% performance, actually got 74% (0.74 C-index)
- Human experts still needed for:
- Data preprocessing and cleaning
- Data representation in suitable format
- Problem formulation as ML task
- Making data ready for AutoML
Impact on Data Science Jobs
- Historical precedent: assembly language to Python didn't eliminate programmers
- Global economy needs more programmers despite automation
- Currently only 5% of global data is analyzed
- AutoML will increase analysis demand
- Businesses/scientists will unlock data value and want more
- AutoML boosts productivity but won't eliminate expert need (foreseeable future)
- Experts still needed for:
- Problem representation and formulation
- Results interpretation
- Correct model application
- Future experts must be more specialized/educated/sophisticated
- Simple tasks (cross-validation, random forest) will be automated with higher quality
- Next generation data scientists must add more value to justify salaries
Future of AutoML
- "Very bright" future predicted
- Era of manual ML scripting reaching critical point
- Prediction: within 5 years, most data analysis will be completely automated
- Or at minimum: AutoML as major part of analysis
- Expert scripting will evolve to:
- Calling AutoML systems
- Configuring AutoML systems
- Customizing AutoML systems
- Higher-level scripting
- Great business opportunities for startups and investments
- Specialized AutoML systems emerging (JADBio caters to bioinformatics)
- Initial proliferation then consolidation to few survivors
- Companies will need embedded AutoML to survive:
- Database vendors
- Data ranking companies
- Cloud providers
- Convert stored data to actionable knowledge automatically
- AutoML embedded in devices (laptops, cars) for behavioral analysis
- Systems will analyze user behavior/usage/environment interaction to become smarter
