AI summary by TwinMind

The inside story of how ChatGPT was built – OpenAI cofounder John Schulman

Dwarkesh Patel

7m 29s

Early LLM Development and Instruction Following

  • OpenAI had base models requiring elaborate prompting with examples for autocomplete functionality
  • Team developed instruction following models to make base models easier to prompt
    • Users could write direct questions instead of complex prompts
    • Deployed in API after GPT-3 as next generation models
  • Concurrent work: Google had Lambda and earlier Mina chatbots
    • More specialized for chat tasks with persona-based interactions
    • Used for fun applications rather than functional tasks like code refactoring

WebGPT and Early Chat Exploration

  • Speaker worked on WebGPT project for question answering with web browsing/retrieval
  • Realized conversational format essential for Q&A due to:
    • Need for follow-up questions
    • Model should ask clarifying questions for ambiguous queries
  • Clear that next version should be conversational

ChatGPT Development Timeline

  • Built conversational chat assistant on GPT-3.5
    • GPT-3.5 training completed beginning of 2022
    • Model showed strong language and coding capabilities
    • Team excited about coding help functionality
  • Worked on project for most of 2022
  • Initially included browsing feature
    • Later deemphasized due to model's strong internal knowledge
    • Browsing wasn't most interesting aspect
  • Beta testing with friends and family before public release

GPT-4 Integration and Model Refinement

  • GPT-4 finished training August 2022
  • Flagship RL effort focused on instruction following
    • These models deployed to production
    • First GPT-4 fine-tunes used instruction following stack
  • Initial GPT-4 results impressive but unreliable:
    • Occasionally amazing outputs
    • Frequent hallucinations
    • Sometimes unhinged responses
    • Not ready for prime time despite obvious quality

Data Integration and Chat Advantages

  • Mixed instruction and chat datasets for best of both worlds
  • Chat models proved superior:
    • Easier to use
    • More sensible automatic behavior
    • Better at knowing own limitations
  • Task definition clarity:
    • Instruction following: "complete text nicely/helpfully" poorly defined
    • Chat: intuitive sense of "helpful robot" behavior
    • Easier for human data labelers to understand expectations
  • Result: more coherent personality and robust sensible behavior

Replicability Discussion

  • Question: Could anyone replicate ChatGPT using public fine-tuning API?
  • Answer: Not exactly, but close approximation possible
  • Requirements for replication:
    • Need GPT-3.5 available for fine-tuning
    • Single iteration insufficient with purely human-written data
    • Multiple iterations needed or iterative supervised fine-tuning
    • Humans editing model-generated outputs essential
    • Hard for models to perfectly fit human-generated data
    • Need RL-like iterative approach
  • Alternative approach: use pre-ChatGPT instruction following model with chat wrapper
    • Would achieve decent approximation
    • That model strong at writing/poetry but weaker at knowing limitations

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome