AI summary by TwinMind

OpenAI’s Deep Research Team on Why Reinforcement Learning is the Future for AI Agents

Sequoia Capital

32m 46s

Product Overview & Origin Story

  • Deep research launched 3 weeks ago, quickly became hit product used by tech luminaries
  • Agent that searches many online websites, creates comprehensive reports
  • Tasks that take humans many hours completed in 5-30 minutes
  • Second agent product from OpenAI after Operator
  • Origin: ~1 year ago, saw success with new reasoning paradigm (training models to think before responding)
  • Initially focused on Math/Science domains, expanded to longer horizon agentic tasks
  • Built by Issa Fulford, Yos Patil, Thomas Dimson
  • Josh Tobin joined 6 months ago after rejoining OpenAI from startup

Target Users & Use Cases

  • Built for anyone doing knowledge work
  • Work applications:
    • Industry analysis, market research
    • Real estate research
    • Scientific research, medical research
    • Coding and code search
    • Finding latest documentation for packages
  • Personal applications:
    • Shopping research (comprehensive product comparisons)
    • Travel planning and recommendations
    • Birthday party planning
    • Car purchasing decisions (release timing research)
  • Surprising use case: Coding became major unexpected application

Business vs Consumer Usage

  • Expected to serve both business and consumer markets
  • Business likely has more tasks requiring extensive research
  • Consumer use cases growing with ChatGPT Plus launch
  • Magic is time savings - hours/days of work reduced to minutes
  • 90% of manual research quality achieved automatically

Technical Architecture

  • Fine-tuned version of O3 (OpenAI's most advanced reasoning model)
  • Trained end-to-end on hard browsing tasks and reasoning tasks
  • Has access to browsing tool and Python tool
  • Trained using reinforcement learning on specific research workflows
  • Model thinks hard about request → searches → reads → understands → decides next search → synthesizes into report with citations

Key Technical Philosophy

  • End-to-end training vs. traditional agent architecture approaches
  • Traditional approach: construct graph of operations with language models as nodes
  • OpenAI approach: train model directly on target tasks without predefined workflow graphs
  • "You get what you optimize for" - core machine learning principle
  • Models become more flexible and creative when trained end-to-end
  • Reinforcement learning tuning critical for most powerful agents

Capabilities & Limitations

  • Excels at: detailed requests requiring extensive internet reading
  • Good at: synthesizing information, finding hard-to-find/obscure facts
  • Can make some new insights from encountered information
  • Not making new scientific discoveries yet
  • Best with specific, detailed descriptions of desired information
  • Strong instruction following with multi-part queries
  • Can format outputs (tables, citations)
  • Underlying model can embed images and create graphs (features coming to ChatGPT)

Quality & Trust Measures

  • Provides citations for all information sources
  • Trained to ensure citation accuracy during training process
  • Still possible for hallucinations or trusting untrustworthy sources
  • Active area for continued improvement

Product Design Decisions

  • Clarification flow: model asks questions before starting research
  • Intentional design - users get best results with detailed, well-specified prompts
  • Not natural user behavior to provide all details upfront
  • Ensures 5-30 minute wait time produces satisfactory, detailed response
  • Users developing workflows: use O1/O1 Pro to refine prompts, then send to Deep Research

Competitive Landscape

  • Multiple "Deep Research" products launched recently
  • OpenAI's differentiation: end-to-end training approach vs. other search products
  • Quality difference comes from training methodology and O3 model capabilities
  • High-quality dataset construction critical to success

Future Roadmap

  • Expand data sources beyond public information to private data
  • Improve browsing and analysis capabilities
  • Integration with broader agent roadmap
  • Recipe scales to more complex tasks: state-of-the-art reasoning model + human tools + direct optimization
  • "AGI is like an operational problem now"

Impact Predictions

  • Sam Altman quote: "will take over single digit percentage of all economically viable tasks"
  • Not labor replacement but time savings tool
  • Knowledge workers could get 1-25% of time back depending on job type
  • Enables tasks previously impossible due to time constraints
  • Particularly impactful for medical use cases (literature review, clinical trial searches)
  • Changes what's possible rather than just efficiency gains

Education & Learning Applications

  • Top use case category
  • Personalized education through AI conversation
  • More efficient and engaging than textbooks
  • AI can personalize information based on user needs/knowledge

2025 Predictions

  • 2025 declared "year of the agent"
  • Reinforcement learning "so back" - timing now right with strong foundation models
  • Yann LeCun cake analogy: unsupervised learning (cake) + supervised learning (frosting) + reinforcement learning (cherries)
  • Previously tried to add cherries before having cake
  • Now have strong pre-trained models ready for RL fine-tuning

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome