AI summary by TwinMind
There's a new best OSS model and it's...weird
Theo - t3․gg
26m 26s
Model Introduction and Claims
- Alibaba released QwQ (32B parameters) - new open-source reasoning model
- Claims performance comparable to DeepSeek R1 (671B parameters)
- Uses reinforcement learning for reasoning capabilities
- Apache 2 license, fully open-weight
- Integrates agent capabilities for tool use and environmental feedback
Benchmark Performance Claims
- QwQ matches DeepSeek R1 on LiveCodeBench and AIM benchmarks
- Outperforms distilled Llama 70B and Qwen 32B models
- Significantly smaller than R1 (32B vs 671B parameters)
- Much easier to deploy than full R1 model
- Grok platform deployed QwQ in minutes vs difficulty with R1
Testing Results - Excessive "Wait" Loops
- QwQ exhibits extreme self-doubt, saying "wait" 262 times on simple prompts
- Comparison: Llama distilled R1 solved same problem much faster
- Fibonacci sequence test: QwQ took 18 "waits" vs DeepSeek R1's 1 "wait"
- Pattern consistent across both Grok deployment and official Qwen chat
- Performance issues caused T3 chat app to lag due to excessive reasoning
UI and Platform Issues
- Qwen chat website built with Svelte, performance problems
- Virtualized scroll container failures, tab freezing
- Date parsing broken, recent chats not displaying correctly
- UI lag from excessive token generation during reasoning loops
- Same issues across different browsers (Firefox, Chrome)
Code Generation Testing
- CSS/HTML generation: Produced basic landing page but with layout bugs
- Tailwind V4 compatibility issues
- Claude 3.5 Sonnet significantly outperformed QwQ on same UI task
- Advent of Code problem: QwQ took 10+ minutes, 61 "waits", 10 "hms"
- Comparison: Llama distilled R1 solved in 34 "waits", 1 "hm"
Fundamental Problems Identified
- Basic counting errors: Inconsistent results counting R's in "strawberry" (2 vs 3)
- Code debugging: Mostly wrong analysis with one correct error identification
- No image capabilities
- Excessive self-gaslighting behavior in reasoning process
- Poor output formatting compared to other reasoning models
Skepticism About Benchmarks
- Performance claims don't match real-world testing experience
- Gap between benchmark scores and practical usability
- "Peak benchmarks vs actual quality improvements" concern
- Model appears optimized for specific benchmark tasks rather than general use
Deployment and Availability
- Available on T3 chat for pro users
- Grok's custom AI chips enable fast inference despite model size
- Much more deployable than full R1 due to smaller parameter count
- Supports distilled R1 models but not full R1
