AI summary by TwinMind

Killer OSS LLM dominating leaderboards, why isn’t anyone talking about Qwen??

Theo - t3․gg

21m 44s

Open Source Model Landscape 2025

  • 2025 wild year for open source models
  • 2024 ended with DeepSeek V3 release
  • 2025 started with DeepSeek R1 - first fully open-source reasoning model
  • Another Chinese model Qwen largely missed despite strong performance
  • Tweet about Qwen only got 90 likes, showing lack of attention

Qwen Performance on Leaderboards

  • Hugging Face Open LLM Leaderboard: all top 10 models are Qwen derivatives
  • Qwen more similar to Llama than R1 but benefits from R1's influence
  • Derivative models concept needs explanation for understanding success

LLM Fundamentals and Derivative Models

  • LLM = collection of weights pointing toward most likely next word
  • Weights determined by training data frequency (e.g., "hello world" vs "hello Jane")
  • Open weight = weights/prioritizations publicly available (sometimes terabytes)
  • Two ways to create derivatives:
    • Fine-tuning: adjust weights with custom inputs
    • Distillation: use larger model (e.g., "Meep seek T1") to generate training data for smaller model ("Glenn")

T3 Chat Model Demonstrations

  • DeepSeek R1 on OpenRouter: full reasoning model with slow inference
  • Llama distilled version on Grok: faster performance, similar responses
  • Qwen distilled (32B parameters): even faster, nearly identical answers
  • Standard Qwen vs distilled Qwen: fundamentally different formatting and behavior
  • Distilled models "hallucinate" reasoning ability by learning from reasoning examples

Qwen's Advantages and Characteristics

  • Very easy to train/fine-tune from other models
  • Open source derivatives allowed due to permissive licensing
  • Extremely efficient: 13kg CO2 among lowest on leaderboard despite top scores
  • Qwen gets into reasoning loops more than other R1 distilled models
  • Burns through loops quickly though

Technical Details and Training

  • Winning leaderboard model: Qwen 2.5 72B base merged with itself, then fine-tuned
  • Qwen 2.5 features:
    • Pre-trained on 18 trillion tokens
    • Better instruction following
    • Improved long text generation
    • Enhanced structured data understanding
    • More resilient to diverse system prompts

DeepSeek R1 Distillation Results

  • DeepSeek released 6 dense models distilled from R1 (Llama and Qwen based)
  • DeepSeek R1 distilled Qwen 32B outperforms O1 mini across benchmarks
  • New state-of-the-art for dense models
  • 1000+ tokens/sec vs OpenAI's ~70-180 tokens/sec (10x+ performance win)
  • Much lower energy usage with very close response quality

OpenAI's Response and Industry Trends

  • Sam Altman roadmap: GPT-4.5 (Orion) will be last non-chain-of-thought model
  • OpenAI concluded reasoning steps improve results enough to include in all models
  • Effectively OpenAI playing catchup to techniques already used by Llama, Qwen, DeepSeek
  • Barrier between quality and cheap/fast options shrinking to months

Licensing Considerations

  • DeepSeek: MIT license (very permissive)
  • Qwen: Apache license (even more permissive, includes patent protection)
  • Llama 3.1/3.3: Custom license with restrictions
    • Must display "built with Llama"
    • >700M monthly users requires Facebook licensing fee
  • Chinese models have most permissive licensing
  • Important business consideration for scaling applications

Future Predictions

  • Qwen likely to remain implementation detail rather than user-facing choice
  • Users may not even know which model they're using (AI-determined routing)
  • Cyclic AI improvement: expensive models train cheaper models to near-equivalent performance
  • Open source ecosystem enabling rapid capability diffusion

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome