AI summary by TwinMind
o3-mini is really good (but does it beat deepseek?)
Theo - t3․gg
22m 34s
O3 Mini Launch Overview
- First reasonably priced and good model from OpenAI
- Release driven by competition with DeepSeek (Chinese company) rather than OpenAI innovation
- Built to compete with Chinese market offerings
- Shows both pricing advantages and implementation quirks
Pricing Analysis Comparison
- GPT-4o: $2.50 input/$10 output per 1M tokens
- GPT-4o Mini: $0.15 input/$0.60 output per 1M tokens
- Offered as free option in T3 chat
- O1: $15 input/$60 output per 1M tokens (expensive, rough to use)
- Claude 3.5 Sonnet: Insanely expensive (biggest T3 chat expense)
- DeepSeek V3: $0.27 input/$1.10 output (new pricing)
- Old V3 pricing: $0.14 input/$0.28 output (hilariously cheap)
- Performance comparable to Claude Sonnet
- DeepSeek R1: $0.55 input/$2.19 output (cheaper than GPT-4o/Sonnet)
- O3 Mini: $1.10 input/$4.40 output
- Incredibly competitive pricing
- Makes O1 not worth touching
- Makes Claude 3.5 Sonnet value proposition weak
- Positions well against DeepSeek R1
Advent of Code Testing Methodology
- Used Christmas-themed programming challenges as model testing
- Tested hardest problems from 2024: Day 17, 21, 24
- Problems that took presenter 1+ hours to solve
- Tested Claude, O1 Pro, O3 Mini, R1 performance
- Goal: evaluate reasoning capabilities on complex programming tasks
Day 17 Results (3-bit Computer Problem)
- Part 1: Build three-bit computer with registers and operations
- O1 Pro: ✓ Correct
- O3 Mini: ✓ Correct
- R1: ✓ Correct
- Claude: ✗ Hallucinated Windows compatibility issues
- Part 2: Find lowest positive value for register A to output copy of itself
- All models failed
- R1 ran indefinitely
- Claude failed due to artificial iteration cap
Day 21 Results (Recursive Robot Keypad Problem)
- Complex nested robot control problem with 3+ layers
- Part 1: All models failed to get correct answer
- Claude ignored file input instruction from prompt
- Part 2: Changed from 3 robots to 25 robots - all models failed
- Presenter's code handled part 2 by just bumping a number
Day 24 Results (Binary Logic Gates Problem)
- Part 1: Combine binary values with AND/OR/XOR operations
- O1 Pro: ✓ Correct (took ~30 minutes)
- O3 Mini: ✓ Correct first time, ✗ failed second attempt
- R1: ✗ Failed completely
- Claude: ✓ Correct with help
- Part 2: All models failed
- O3 Mini hung indefinitely when given combined part 1+2 prompt
Model Performance Insights
- Only OpenAI reasoning models (O1/O3) solved Day 24
- High failure rate even for reasoning models
- R1 surprisingly failed Day 24 completely
- Difference between O-series and R1 models apparent
- Claude performed worst overall, ignored instructions
UI/UX Issues with OpenAI
- O3 Mini prompts hang for 30+ minutes then fail silently
- ChatGPT UI extremely laggy (half-second delays)
- Terrible scroll behavior and rendering issues
- Queries can take 20+ minutes, fail if tab left alone
- Pagination and scroll containers broken
- T3 chat has none of these problems
R1 Distilled Model Performance
- Hosted on Grok, extremely fast reasoning output
- Not full R1 reasoning model, distilled from R1 into Llama
- Behaves similarly but works like traditional LLM
- Outputs reasoning text based on R1 training
- Failed same problems as other models but much faster
O3 Mini Technical Limitations
- No reasoning data exposed via API (only in ChatGPT UI)
- Prevents companies like DeepSeek from training on reasoning data
- No response streaming - sends complete response at once
- Poor user experience compared to streaming models
- Different reasoning effort levels: Low/Medium/High
Formatting Issues
- O3 Mini uses entirely different output format
- Doesn't use standard markdown formatting
- Adds weird bars everywhere
- Doesn't indicate programming language in code blocks
- System prompts don't fix formatting issues
- Makes UI integration difficult
- Particularly bad for PHP developers
Competitive Analysis
- OpenAI throwing shots at DeepSeek distillation methods
- Mck Wrigley's agents moved from O1 to O3 Mini
- Working as well or better than before
- 9x cheaper and significantly faster
- CodeForces scores show O3 Mini beating O1 across all power levels
- Reasoning model now cheaper than non-reasoning models
Market Impact
- O3 Mini exactly 2x price of GPT-4o Mini (intentional pricing)
- Prevents easy price competition from self-hosted DeepSeek
- American reasoning model cheaper than non-reasoning model
- Response to DeepSeek market disruption rather than innovation
DeepSeek Advantages Still Present
- R1 reveals reasoning process (not hidden)
- Available via API and self-hosting
- Way cheaper than O3 Mini
- Open-source model flexibility
- Geopolitical implications for AI development
Overall Assessment
- O3 Mini: Good model, keeps OpenAI competitive
- Not revolutionary but result of DeepSeek revolution
- Attempt to catch up to marketplace disruption
- Exciting for pricing and accessibility
- Still lacks transparency and openness of DeepSeek approach
