AI summary by TwinMind
OpenAI’s Deep Research Team on Why Reinforcement Learning is the Future for AI Agents
Sequoia Capital
32m 46s
Product Overview & Origin Story
- Deep research launched 3 weeks ago, quickly became hit product used by tech luminaries
- Agent that searches many online websites, creates comprehensive reports
- Tasks that take humans many hours completed in 5-30 minutes
- Second agent product from OpenAI after Operator
- Origin: ~1 year ago, saw success with new reasoning paradigm (training models to think before responding)
- Initially focused on Math/Science domains, expanded to longer horizon agentic tasks
- Built by Issa Fulford, Yos Patil, Thomas Dimson
- Josh Tobin joined 6 months ago after rejoining OpenAI from startup
Target Users & Use Cases
- Built for anyone doing knowledge work
- Work applications:
- Industry analysis, market research
- Real estate research
- Scientific research, medical research
- Coding and code search
- Finding latest documentation for packages
- Personal applications:
- Shopping research (comprehensive product comparisons)
- Travel planning and recommendations
- Birthday party planning
- Car purchasing decisions (release timing research)
- Surprising use case: Coding became major unexpected application
Business vs Consumer Usage
- Expected to serve both business and consumer markets
- Business likely has more tasks requiring extensive research
- Consumer use cases growing with ChatGPT Plus launch
- Magic is time savings - hours/days of work reduced to minutes
- 90% of manual research quality achieved automatically
Technical Architecture
- Fine-tuned version of O3 (OpenAI's most advanced reasoning model)
- Trained end-to-end on hard browsing tasks and reasoning tasks
- Has access to browsing tool and Python tool
- Trained using reinforcement learning on specific research workflows
- Model thinks hard about request → searches → reads → understands → decides next search → synthesizes into report with citations
Key Technical Philosophy
- End-to-end training vs. traditional agent architecture approaches
- Traditional approach: construct graph of operations with language models as nodes
- OpenAI approach: train model directly on target tasks without predefined workflow graphs
- "You get what you optimize for" - core machine learning principle
- Models become more flexible and creative when trained end-to-end
- Reinforcement learning tuning critical for most powerful agents
Capabilities & Limitations
- Excels at: detailed requests requiring extensive internet reading
- Good at: synthesizing information, finding hard-to-find/obscure facts
- Can make some new insights from encountered information
- Not making new scientific discoveries yet
- Best with specific, detailed descriptions of desired information
- Strong instruction following with multi-part queries
- Can format outputs (tables, citations)
- Underlying model can embed images and create graphs (features coming to ChatGPT)
Quality & Trust Measures
- Provides citations for all information sources
- Trained to ensure citation accuracy during training process
- Still possible for hallucinations or trusting untrustworthy sources
- Active area for continued improvement
Product Design Decisions
- Clarification flow: model asks questions before starting research
- Intentional design - users get best results with detailed, well-specified prompts
- Not natural user behavior to provide all details upfront
- Ensures 5-30 minute wait time produces satisfactory, detailed response
- Users developing workflows: use O1/O1 Pro to refine prompts, then send to Deep Research
Competitive Landscape
- Multiple "Deep Research" products launched recently
- OpenAI's differentiation: end-to-end training approach vs. other search products
- Quality difference comes from training methodology and O3 model capabilities
- High-quality dataset construction critical to success
Future Roadmap
- Expand data sources beyond public information to private data
- Improve browsing and analysis capabilities
- Integration with broader agent roadmap
- Recipe scales to more complex tasks: state-of-the-art reasoning model + human tools + direct optimization
- "AGI is like an operational problem now"
Impact Predictions
- Sam Altman quote: "will take over single digit percentage of all economically viable tasks"
- Not labor replacement but time savings tool
- Knowledge workers could get 1-25% of time back depending on job type
- Enables tasks previously impossible due to time constraints
- Particularly impactful for medical use cases (literature review, clinical trial searches)
- Changes what's possible rather than just efficiency gains
Education & Learning Applications
- Top use case category
- Personalized education through AI conversation
- More efficient and engaging than textbooks
- AI can personalize information based on user needs/knowledge
2025 Predictions
- 2025 declared "year of the agent"
- Reinforcement learning "so back" - timing now right with strong foundation models
- Yann LeCun cake analogy: unsupervised learning (cake) + supervised learning (frosting) + reinforcement learning (cherries)
- Previously tried to add cherries before having cake
- Now have strong pre-trained models ready for RL fine-tuning
