AI summary by TwinMind

What's next for AI agentic workflows ft. Andrew Ng of AI Fund

Sequoia Capital

13m 40s

Introduction and Background

  • Andrew Ng introduced as Stanford CS professor, early neural networks with GPUs developer
  • Creator of Coursera, deeplearning.ai courses
  • Founder and early lead of Google Brain
  • Humorous exchange about CS229 problem set grade from 10 years ago

Agentic vs Non-Agentic Workflows

  • Non-agentic workflow: Type prompt → generate answer (like asking person to write essay start-to-finish without backspace)
  • Agentic workflow: Iterative process
    • Write essay outline
    • Do web research if needed
    • Write first draft
    • Read and revise draft
    • Multiple iterations of thinking and revision
  • Agentic workflows deliver "remarkably better results"

Case Study: Human Eval Benchmark

  • Coding benchmark: Problems like "given non-empty list of integers, return sum of elements at even positions"
  • Performance comparison:
    • GPT-3.5 zero shot: 48% correct
    • GPT-4 zero shot: 67% correct
    • GPT-3.5 with agentic workflow: outperforms GPT-4
    • GPT-4 with agentic workflow: performs very well
  • Key insight: GPT-3.5 + agentic workflow > GPT-4 zero shot
  • Significant consequences for application building approach

Four Agent Design Patterns

1. Reflection (Robust Technology)

  • Process:
    • Coder agent writes code for task
    • Same LM prompted to review its own code: "check code carefully for correctness, efficiency, good construction"
    • LM spots problems (e.g., "bug in line 5")
    • Generates improved version 2
  • Extensions:
    • Run unit tests, analyze failures
    • Iterative conversation about test failures
    • Generate version 3 based on feedback
  • Multi-agent evolution: Separate coder agent and critic agent (same base LM, different prompts)

2. Tool Use

  • Examples: GitHub Copilot, GPT-4 capabilities
  • Common tools: Web search, code generation/execution, analysis, information gathering, personal productivity
  • Historical note: Much early tool use work originated from computer vision community
    • Pre-multimodal LMs were "blind to images"
    • Required function calls to manipulate images (generation, object detection)
  • Expands LM capabilities significantly

3. Planning (Emerging Technology)

  • Reliability: Sometimes mind-blowing results, but not always reliable
  • Capabilities: AI agents can reroute around failures autonomously
  • Example workflow (adapted from Hugging GPT paper):
    • Task: Generate image of girl reading book in same pose as boy in example image
    • Steps: Determine boy's pose → find pose extraction model → find pose image synthesis model → generate image → image-to-text → text-to-speech
  • Current usage: Research agents for preliminary work
    • Delegate research tasks, return in few minutes
    • Works sometimes, part of personal workflow

4. Multi-Agent Collaboration

  • ChatDev example: Open source, runs on laptop
  • Agent roles: CEO, software engineer, designer, product manager, tester
  • Process: Single LM prompted to act as different roles in extended conversation
  • Capabilities: Develop complex programs (e.g., "develop a GO game")
    • Spend minutes writing code, testing, iterating
    • Generate surprisingly complex programs
  • Multi-agent debate: Different agents (ChatGPT vs Gemini) debating improves performance

Future Implications and Trends

Workflow Changes

  • Expectation shift: Need to wait minutes/hours for agent responses vs instant feedback
  • Analogy: Like novice managers checking on delegated work after 5 minutes (unproductive)
  • Patience required: Delegate tasks to AI agents and wait appropriately

Fast Token Generation Importance

  • Reason: Agentic workflows require multiple iterations
  • LM-to-LM communication: Generating tokens faster than humans can read
  • Trade-off hypothesis: Fast tokens from slightly lower quality LM might outperform slower tokens from better LM
  • Controversial: Allows more loop iterations (like GPT-3.5 + agentic results)

Model Expectations

  • Upcoming models: Looking forward to Claude 5, Claude 4, GPT-5, Gemini 2.0
  • Performance prediction: Agentic reasoning with current models might approach GPT-5 zero-shot performance on some applications
  • Path to AGI: Journey rather than destination, agentic workflows represent small step forward

Recommended Reading

  • References provided for each of the four design pattern sections
  • Emphasis on learning more about these technologies

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome