AI summary by TwinMind
The inside story of how ChatGPT was built – OpenAI cofounder John Schulman
Dwarkesh Patel
7m 29s
Early LLM Development and Instruction Following
- OpenAI had base models requiring elaborate prompting with examples for autocomplete functionality
- Team developed instruction following models to make base models easier to prompt
- Users could write direct questions instead of complex prompts
- Deployed in API after GPT-3 as next generation models
- Concurrent work: Google had Lambda and earlier Mina chatbots
- More specialized for chat tasks with persona-based interactions
- Used for fun applications rather than functional tasks like code refactoring
WebGPT and Early Chat Exploration
- Speaker worked on WebGPT project for question answering with web browsing/retrieval
- Realized conversational format essential for Q&A due to:
- Need for follow-up questions
- Model should ask clarifying questions for ambiguous queries
- Clear that next version should be conversational
ChatGPT Development Timeline
- Built conversational chat assistant on GPT-3.5
- GPT-3.5 training completed beginning of 2022
- Model showed strong language and coding capabilities
- Team excited about coding help functionality
- Worked on project for most of 2022
- Initially included browsing feature
- Later deemphasized due to model's strong internal knowledge
- Browsing wasn't most interesting aspect
- Beta testing with friends and family before public release
GPT-4 Integration and Model Refinement
- GPT-4 finished training August 2022
- Flagship RL effort focused on instruction following
- These models deployed to production
- First GPT-4 fine-tunes used instruction following stack
- Initial GPT-4 results impressive but unreliable:
- Occasionally amazing outputs
- Frequent hallucinations
- Sometimes unhinged responses
- Not ready for prime time despite obvious quality
Data Integration and Chat Advantages
- Mixed instruction and chat datasets for best of both worlds
- Chat models proved superior:
- Easier to use
- More sensible automatic behavior
- Better at knowing own limitations
- Task definition clarity:
- Instruction following: "complete text nicely/helpfully" poorly defined
- Chat: intuitive sense of "helpful robot" behavior
- Easier for human data labelers to understand expectations
- Result: more coherent personality and robust sensible behavior
Replicability Discussion
- Question: Could anyone replicate ChatGPT using public fine-tuning API?
- Answer: Not exactly, but close approximation possible
- Requirements for replication:
- Need GPT-3.5 available for fine-tuning
- Single iteration insufficient with purely human-written data
- Multiple iterations needed or iterative supervised fine-tuning
- Humans editing model-generated outputs essential
- Hard for models to perfectly fit human-generated data
- Need RL-like iterative approach
- Alternative approach: use pre-ChatGPT instruction following model with chat wrapper
- Would achieve decent approximation
- That model strong at writing/poetry but weaker at knowing limitations
