AI summary by TwinMind

There's a new best OSS model and it's...weird

Theo - t3․gg

26m 26s

Model Introduction and Claims

  • Alibaba released QwQ (32B parameters) - new open-source reasoning model
  • Claims performance comparable to DeepSeek R1 (671B parameters)
  • Uses reinforcement learning for reasoning capabilities
  • Apache 2 license, fully open-weight
  • Integrates agent capabilities for tool use and environmental feedback

Benchmark Performance Claims

  • QwQ matches DeepSeek R1 on LiveCodeBench and AIM benchmarks
  • Outperforms distilled Llama 70B and Qwen 32B models
  • Significantly smaller than R1 (32B vs 671B parameters)
  • Much easier to deploy than full R1 model
  • Grok platform deployed QwQ in minutes vs difficulty with R1

Testing Results - Excessive "Wait" Loops

  • QwQ exhibits extreme self-doubt, saying "wait" 262 times on simple prompts
  • Comparison: Llama distilled R1 solved same problem much faster
  • Fibonacci sequence test: QwQ took 18 "waits" vs DeepSeek R1's 1 "wait"
  • Pattern consistent across both Grok deployment and official Qwen chat
  • Performance issues caused T3 chat app to lag due to excessive reasoning

UI and Platform Issues

  • Qwen chat website built with Svelte, performance problems
  • Virtualized scroll container failures, tab freezing
  • Date parsing broken, recent chats not displaying correctly
  • UI lag from excessive token generation during reasoning loops
  • Same issues across different browsers (Firefox, Chrome)

Code Generation Testing

  • CSS/HTML generation: Produced basic landing page but with layout bugs
  • Tailwind V4 compatibility issues
  • Claude 3.5 Sonnet significantly outperformed QwQ on same UI task
  • Advent of Code problem: QwQ took 10+ minutes, 61 "waits", 10 "hms"
  • Comparison: Llama distilled R1 solved in 34 "waits", 1 "hm"

Fundamental Problems Identified

  • Basic counting errors: Inconsistent results counting R's in "strawberry" (2 vs 3)
  • Code debugging: Mostly wrong analysis with one correct error identification
  • No image capabilities
  • Excessive self-gaslighting behavior in reasoning process
  • Poor output formatting compared to other reasoning models

Skepticism About Benchmarks

  • Performance claims don't match real-world testing experience
  • Gap between benchmark scores and practical usability
  • "Peak benchmarks vs actual quality improvements" concern
  • Model appears optimized for specific benchmark tasks rather than general use

Deployment and Availability

  • Available on T3 chat for pro users
  • Grok's custom AI chips enable fast inference despite model size
  • Much more deployable than full R1 due to smaller parameter count
  • Supports distilled R1 models but not full R1

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome