AI summary by TwinMind

OpenAI Discovers JSON (And Zod???)

Theo - t3․gg

28m 30s

OpenAI Structured Outputs Announcement

  • OpenAI introduces structured outputs in API - model outputs now reliably adhere to developer-supplied JSON schema
  • Previously JSON mode improved reliability but didn't guarantee schema conformance
  • New feature ensures model-generated outputs exactly match provided JSON schemas
  • Major improvement from previous ~40% schema compliance to 100% with structured outputs

Technical Implementation Details

  • Uses constrained sampling/constrained decoding technique
  • Restricts model to only tokens valid according to supplied schema
  • Converts JSON schema into context-free grammar (CFG)
  • Pre-processes schema components for efficient sampling
  • First request with new schema incurs latency penalty (up to 10 seconds, complex schemas up to 1 minute)
  • Subsequent responses are fast due to cached CFG
  • Dynamic constraint decoding determines valid tokens after each generation

New GPT-4 Model Release

  • GPT-4o-2024-08-06 model specifically trained for structured outputs
  • 50% savings on input costs ($2.50 per million tokens)
  • 33% savings on output costs ($10 per million tokens)
  • Trained to understand complex schemas and produce matching outputs
  • Even with training improvements, only achieved 93% reliability without constraints

Zod Integration and Developer Experience

  • Native Zod support in OpenAI SDK via openai/helpers/zod
  • Can use Zod objects as schema definitions
  • Pydantic support for Python developers
  • Structured outputs available via tools (setting strict: true)
  • Response format support with client.beta.chat.completions.parse()

Use Cases and Examples

  • Dynamic UI generation based on user intent
  • Extracting structured data from unstructured inputs
  • Action item extraction from meeting notes with due dates/owners
  • Separating reasoning from final answers using chain-of-thought fields
  • Math problem solving with step-by-step explanations

Limitations and Constraints

  • Subset of JSON schema supported (no min/max length, patterns, formats)
  • No minimum/maximum for numbers or arrays
  • First API response with new schema has additional latency
  • Model can fail if refusing unsafe requests (refusal boolean returned)
  • Can fail if hitting max tokens before completion
  • Not compatible with parallel function calls
  • Schemas not eligible for zero data retention

Vercel AI SDK Response

  • Vercel AI SDK 3.3 released same day with competing features
  • Added tracing with DataDog, Axiom, Sentry
  • Multimodal file attachments with useChat
  • Stream object generation with useObject hook
  • Structured object generation for tools
  • Uses describe() in Zod schemas for better prompting
  • Streaming structured objects directly to client
  • Example shows expense categorization with real-time streaming

Competitive Landscape Analysis

  • OpenAI catching up to Vercel's developer experience improvements
  • Vercel historically made AI consumption easier for web developers
  • OpenAI addressing vendor lock-in concerns (easy switching between providers)
  • Vercel leap-frogging with advanced streaming and client-side features
  • Both embracing TypeScript ecosystem and Zod as standard

Research Context

  • Study showing format restrictions decrease LLM reasoning ability
  • Suggested two-step approach: unconstrained reasoning → formatting with smaller model
  • OpenAI's approach may use similar multi-step process internally
  • Focus on developer experience (DX) improvements from OpenAI

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome