AI summary by TwinMind
What's next for AI agentic workflows ft. Andrew Ng of AI Fund
Sequoia Capital
13m 40s
Introduction and Background
- Andrew Ng introduced as Stanford CS professor, early neural networks with GPUs developer
- Creator of Coursera, deeplearning.ai courses
- Founder and early lead of Google Brain
- Humorous exchange about CS229 problem set grade from 10 years ago
Agentic vs Non-Agentic Workflows
- Non-agentic workflow: Type prompt → generate answer (like asking person to write essay start-to-finish without backspace)
- Agentic workflow: Iterative process
- Write essay outline
- Do web research if needed
- Write first draft
- Read and revise draft
- Multiple iterations of thinking and revision
- Agentic workflows deliver "remarkably better results"
Case Study: Human Eval Benchmark
- Coding benchmark: Problems like "given non-empty list of integers, return sum of elements at even positions"
- Performance comparison:
- GPT-3.5 zero shot: 48% correct
- GPT-4 zero shot: 67% correct
- GPT-3.5 with agentic workflow: outperforms GPT-4
- GPT-4 with agentic workflow: performs very well
- Key insight: GPT-3.5 + agentic workflow > GPT-4 zero shot
- Significant consequences for application building approach
Four Agent Design Patterns
1. Reflection (Robust Technology)
- Process:
- Coder agent writes code for task
- Same LM prompted to review its own code: "check code carefully for correctness, efficiency, good construction"
- LM spots problems (e.g., "bug in line 5")
- Generates improved version 2
- Extensions:
- Run unit tests, analyze failures
- Iterative conversation about test failures
- Generate version 3 based on feedback
- Multi-agent evolution: Separate coder agent and critic agent (same base LM, different prompts)
2. Tool Use
- Examples: GitHub Copilot, GPT-4 capabilities
- Common tools: Web search, code generation/execution, analysis, information gathering, personal productivity
- Historical note: Much early tool use work originated from computer vision community
- Pre-multimodal LMs were "blind to images"
- Required function calls to manipulate images (generation, object detection)
- Expands LM capabilities significantly
3. Planning (Emerging Technology)
- Reliability: Sometimes mind-blowing results, but not always reliable
- Capabilities: AI agents can reroute around failures autonomously
- Example workflow (adapted from Hugging GPT paper):
- Task: Generate image of girl reading book in same pose as boy in example image
- Steps: Determine boy's pose → find pose extraction model → find pose image synthesis model → generate image → image-to-text → text-to-speech
- Current usage: Research agents for preliminary work
- Delegate research tasks, return in few minutes
- Works sometimes, part of personal workflow
4. Multi-Agent Collaboration
- ChatDev example: Open source, runs on laptop
- Agent roles: CEO, software engineer, designer, product manager, tester
- Process: Single LM prompted to act as different roles in extended conversation
- Capabilities: Develop complex programs (e.g., "develop a GO game")
- Spend minutes writing code, testing, iterating
- Generate surprisingly complex programs
- Multi-agent debate: Different agents (ChatGPT vs Gemini) debating improves performance
Future Implications and Trends
Workflow Changes
- Expectation shift: Need to wait minutes/hours for agent responses vs instant feedback
- Analogy: Like novice managers checking on delegated work after 5 minutes (unproductive)
- Patience required: Delegate tasks to AI agents and wait appropriately
Fast Token Generation Importance
- Reason: Agentic workflows require multiple iterations
- LM-to-LM communication: Generating tokens faster than humans can read
- Trade-off hypothesis: Fast tokens from slightly lower quality LM might outperform slower tokens from better LM
- Controversial: Allows more loop iterations (like GPT-3.5 + agentic results)
Model Expectations
- Upcoming models: Looking forward to Claude 5, Claude 4, GPT-5, Gemini 2.0
- Performance prediction: Agentic reasoning with current models might approach GPT-5 zero-shot performance on some applications
- Path to AGI: Journey rather than destination, agentic workflows represent small step forward
Recommended Reading
- References provided for each of the four design pattern sections
- Emphasis on learning more about these technologies
