Open Source Model Landscape 2025

• 2025 wild year for open source models
• 2024 ended with DeepSeek V3 release
• 2025 started with DeepSeek R1 - first fully open-source reasoning model
• Another Chinese model Qwen largely missed despite strong performance
• Tweet about Qwen only got 90 likes, showing lack of attention

Qwen Performance on Leaderboards

• Hugging Face Open LLM Leaderboard: all top 10 models are Qwen derivatives
• Qwen more similar to Llama than R1 but benefits from R1's influence
• Derivative models concept needs explanation for understanding success

LLM Fundamentals and Derivative Models

• LLM = collection of weights pointing toward most likely next word
• Weights determined by training data frequency (e.g., "hello world" vs "hello Jane")
• Open weight = weights/prioritizations publicly available (sometimes terabytes)
• Two ways to create derivatives:
  • Fine-tuning: adjust weights with custom inputs
  • Distillation: use larger model (e.g., "Meep seek T1") to generate training data for smaller model ("Glenn")

T3 Chat Model Demonstrations

• DeepSeek R1 on OpenRouter: full reasoning model with slow inference
• Llama distilled version on Grok: faster performance, similar responses
• Qwen distilled (32B parameters): even faster, nearly identical answers
• Standard Qwen vs distilled Qwen: fundamentally different formatting and behavior
• Distilled models "hallucinate" reasoning ability by learning from reasoning examples

Qwen's Advantages and Characteristics

• Very easy to train/fine-tune from other models
• Open source derivatives allowed due to permissive licensing
• Extremely efficient: 13kg CO2 among lowest on leaderboard despite top scores
• Qwen gets into reasoning loops more than other R1 distilled models
• Burns through loops quickly though

Technical Details and Training

• Winning leaderboard model: Qwen 2.5 72B base merged with itself, then fine-tuned
• Qwen 2.5 features:
  • Pre-trained on 18 trillion tokens
  • Better instruction following
  • Improved long text generation
  • Enhanced structured data understanding
  • More resilient to diverse system prompts

DeepSeek R1 Distillation Results

• DeepSeek released 6 dense models distilled from R1 (Llama and Qwen based)
• DeepSeek R1 distilled Qwen 32B outperforms O1 mini across benchmarks
• New state-of-the-art for dense models
• 1000+ tokens/sec vs OpenAI's ~70-180 tokens/sec (10x+ performance win)
• Much lower energy usage with very close response quality

OpenAI's Response and Industry Trends

• Sam Altman roadmap: GPT-4.5 (Orion) will be last non-chain-of-thought model
• OpenAI concluded reasoning steps improve results enough to include in all models
• Effectively OpenAI playing catchup to techniques already used by Llama, Qwen, DeepSeek
• Barrier between quality and cheap/fast options shrinking to months

Licensing Considerations

• DeepSeek: MIT license (very permissive)
• Qwen: Apache license (even more permissive, includes patent protection)
• Llama 3.1/3.3: Custom license with restrictions
  • Must display "built with Llama"
  • >700M monthly users requires Facebook licensing fee
• Chinese models have most permissive licensing
• Important business consideration for scaling applications

Future Predictions

• Qwen likely to remain implementation detail rather than user-facing choice
• Users may not even know which model they're using (AI-determined routing)
• Cyclic AI improvement: expensive models train cheaper models to near-equivalent performance
• Open source ecosystem enabling rapid capability diffusion