AI summary by TwinMind

o3-mini is really good (but does it beat deepseek?)

Theo - t3․gg

22m 34s

O3 Mini Launch Overview

  • First reasonably priced and good model from OpenAI
  • Release driven by competition with DeepSeek (Chinese company) rather than OpenAI innovation
  • Built to compete with Chinese market offerings
  • Shows both pricing advantages and implementation quirks

Pricing Analysis Comparison

  • GPT-4o: $2.50 input/$10 output per 1M tokens
  • GPT-4o Mini: $0.15 input/$0.60 output per 1M tokens
    • Offered as free option in T3 chat
  • O1: $15 input/$60 output per 1M tokens (expensive, rough to use)
  • Claude 3.5 Sonnet: Insanely expensive (biggest T3 chat expense)
  • DeepSeek V3: $0.27 input/$1.10 output (new pricing)
    • Old V3 pricing: $0.14 input/$0.28 output (hilariously cheap)
    • Performance comparable to Claude Sonnet
  • DeepSeek R1: $0.55 input/$2.19 output (cheaper than GPT-4o/Sonnet)
  • O3 Mini: $1.10 input/$4.40 output
    • Incredibly competitive pricing
    • Makes O1 not worth touching
    • Makes Claude 3.5 Sonnet value proposition weak
    • Positions well against DeepSeek R1

Advent of Code Testing Methodology

  • Used Christmas-themed programming challenges as model testing
  • Tested hardest problems from 2024: Day 17, 21, 24
  • Problems that took presenter 1+ hours to solve
  • Tested Claude, O1 Pro, O3 Mini, R1 performance
  • Goal: evaluate reasoning capabilities on complex programming tasks

Day 17 Results (3-bit Computer Problem)

  • Part 1: Build three-bit computer with registers and operations
    • O1 Pro: ✓ Correct
    • O3 Mini: ✓ Correct
    • R1: ✓ Correct
    • Claude: ✗ Hallucinated Windows compatibility issues
  • Part 2: Find lowest positive value for register A to output copy of itself
    • All models failed
    • R1 ran indefinitely
    • Claude failed due to artificial iteration cap

Day 21 Results (Recursive Robot Keypad Problem)

  • Complex nested robot control problem with 3+ layers
  • Part 1: All models failed to get correct answer
  • Claude ignored file input instruction from prompt
  • Part 2: Changed from 3 robots to 25 robots - all models failed
  • Presenter's code handled part 2 by just bumping a number

Day 24 Results (Binary Logic Gates Problem)

  • Part 1: Combine binary values with AND/OR/XOR operations
    • O1 Pro: ✓ Correct (took ~30 minutes)
    • O3 Mini: ✓ Correct first time, ✗ failed second attempt
    • R1: ✗ Failed completely
    • Claude: ✓ Correct with help
  • Part 2: All models failed
  • O3 Mini hung indefinitely when given combined part 1+2 prompt

Model Performance Insights

  • Only OpenAI reasoning models (O1/O3) solved Day 24
  • High failure rate even for reasoning models
  • R1 surprisingly failed Day 24 completely
  • Difference between O-series and R1 models apparent
  • Claude performed worst overall, ignored instructions

UI/UX Issues with OpenAI

  • O3 Mini prompts hang for 30+ minutes then fail silently
  • ChatGPT UI extremely laggy (half-second delays)
  • Terrible scroll behavior and rendering issues
  • Queries can take 20+ minutes, fail if tab left alone
  • Pagination and scroll containers broken
  • T3 chat has none of these problems

R1 Distilled Model Performance

  • Hosted on Grok, extremely fast reasoning output
  • Not full R1 reasoning model, distilled from R1 into Llama
  • Behaves similarly but works like traditional LLM
  • Outputs reasoning text based on R1 training
  • Failed same problems as other models but much faster

O3 Mini Technical Limitations

  • No reasoning data exposed via API (only in ChatGPT UI)
  • Prevents companies like DeepSeek from training on reasoning data
  • No response streaming - sends complete response at once
  • Poor user experience compared to streaming models
  • Different reasoning effort levels: Low/Medium/High

Formatting Issues

  • O3 Mini uses entirely different output format
  • Doesn't use standard markdown formatting
  • Adds weird bars everywhere
  • Doesn't indicate programming language in code blocks
  • System prompts don't fix formatting issues
  • Makes UI integration difficult
  • Particularly bad for PHP developers

Competitive Analysis

  • OpenAI throwing shots at DeepSeek distillation methods
  • Mck Wrigley's agents moved from O1 to O3 Mini
    • Working as well or better than before
    • 9x cheaper and significantly faster
  • CodeForces scores show O3 Mini beating O1 across all power levels
  • Reasoning model now cheaper than non-reasoning models

Market Impact

  • O3 Mini exactly 2x price of GPT-4o Mini (intentional pricing)
  • Prevents easy price competition from self-hosted DeepSeek
  • American reasoning model cheaper than non-reasoning model
  • Response to DeepSeek market disruption rather than innovation

DeepSeek Advantages Still Present

  • R1 reveals reasoning process (not hidden)
  • Available via API and self-hosting
  • Way cheaper than O3 Mini
  • Open-source model flexibility
  • Geopolitical implications for AI development

Overall Assessment

  • O3 Mini: Good model, keeps OpenAI competitive
  • Not revolutionary but result of DeepSeek revolution
  • Attempt to catch up to marketplace disruption
  • Exciting for pricing and accessibility
  • Still lacks transparency and openness of DeepSeek approach

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome