AI summary by TwinMind
A Measured Take on Devin
Theo - t3․gg
44m 41s
Initial Reaction to Devon AI Announcement
- Cognition Labs posted video on Twitter causing massive Dev World buzz
- Half of speaker's feed/DMs about Devon AI software engineer
- Devon claims: first AI software engineer, state-of-the-art on SWE-Bench coding benchmark
- Passed practical engineering interviews from leading AI companies
- Completed real jobs on Upwork
- Speaker notes engineers exist who meet these criteria but still perform poorly
SWE-Bench Performance Analysis
- Devon resolves 13-14% of GitHub issues without assistance on SWE-Bench
- Previous models: 1.9-2% unassisted, up to 5% assisted
- Tool covering only 14% of issues "isn't as good as an engineer"
- Speaker remains curious but not impressed by initial claims
Demo Video Breakdown - API Benchmarking
- Devon asked to benchmark llama performance on different API providers
- Creates step-by-step plan, uses command line, code editor, browser
- AI opens documentation to learn APIs autonomously
- Chat interface shows human providing guidance throughout (hyper-speed typing noted)
- Human inputs: "measure wall clock run time of API," "use REST API directly," "build whole project using only metric we care about is tokens per second"
- Devon encounters unexpected error, adds debugging print statement, reruns code
- Builds and deploys website with full styling as visualization
- Speaker criticizes poor video editing quality affecting retention
Game of Life Implementation Analysis
- Devon deployed functional Game of Life on Netlify
- Console errors present: "prevent defaults not a function of e-mouse drag"
- Time analysis: 1:40-1:41 initial creation, 1:41-1:51 production build (10 minutes total)
- Subsequent changes: 7+ minutes per modification
- Bug fix: 3 minutes (1:53-2:11)
- Frame rate changes: 4 minutes
- Responsive design + interactivity: additional time
- Speaker notes 10+ minutes per change is problematic for iteration
Blog Post Learning Demo
- Sarah demonstrates Devon learning from blog post to generate desktop background images
- Time stamps: 12:59 start to 1:41 completion = 42 minutes
- Generated one good image (Sarah) and two difficult-to-read bonus images
- Speaker notes: similar time to decent engineer but without retained knowledge
- AI must retrain for each similar task vs. human learning persistence
Bug Fix Demo - Senpai Repository
- Devon fixes log function bug in Python algebra system
- Total time: 1:16-12:12 = 1 hour
- Breakdown: 12 minutes setup/bug identification, 30-40 minutes debugging/fixing, 30 minutes testing/cleanup
- Issue: integer division causing log(0) = infinity
- Fixed by replacing integer division with true division
- Speaker questions if test suite was slow or if AI process was inefficient
Audio Quality Criticism
- Multiple demos have poor audio quality
- Speaker demonstrates Adobe audio enhancement tool
- Criticizes lack of production quality for 500K+ impression videos
- Suggests using AI for presentation improvement rather than code generation
Skepticism Points
- No live typing demonstrations - all pre-recorded scrolling through completed work
- Cognition hiding behind waitlist rather than public access
- Comparison to OpenAI's Sora transparency about limitations
- OpenAI shows failure cases and current weaknesses openly
Technical Architecture Analysis
- Bloomberg article reveals: combines LLMs (GPT-4) with reinforcement learning
- Not building own models, just wrapping GPT-4
- 3 million token context claimed (vs GPT-4.5 rumored 256K = 12x difference)
- Speaker notes this approach easily replicable by competitors in 3-6 months
- Company raised $21 million from Peter Thiel's Founders Fund
- 10-person team, incorporated only 2 months ago
Founder Background
- Woo Brothers: world-renowned competitive programmers since teenagers
- Helped elevate US national coding team against Chinese/Eastern European rivals
- One brother (Yan) at Harvard, hasn't told parents about dropping out
- Previous experience at Scale AI (noted as "more early than talented")
- CEO demonstrated exceptional math competition skills as child
Real-World Testing Results
- Matt Bielman (Netlify CEO) built todo app with Devon
- App has multiple bugs: add task button doesn't work, no local storage, dark mode toggle breaks
- Ryan Carniato (Solid JS creator) analysis: 130KB JavaScript bundle, ~80 Lighthouse score
- Uses React + Chakra UI + Framer Motion for simple todo app
- 26x larger than equivalent implementations in Svelte/Preact/Solid (5KB vs 130KB)
- Described as "confidently mid" - replicating intern-level work
GitHub Analysis
- Devon has GitHub account with rust code contributions
- Code quality issues: awful comments, default exit codes for errors
- Accidentally includes test data in production changes
- Represents realistic developer mistakes but not professional quality
Security Concerns
- Generated code contains security vulnerabilities
- Uses insecure authentication patterns that bypass layout checks
- Doesn't use modern React patterns (React Query, proper state management)
- Race conditions present in async operations
Comparison to Competitors
- Magic AI raised $100M, building own models (more legitimate approach)
- Marius (Y Combinator company) - honest about limitations, publicly available
- Marius focuses on scaffolding rather than claiming to replace engineers
- Speaker invested in Marius due to founder transparency and realistic scope
Cost and Efficiency Analysis
- 40+ minutes of GPU-heavy computation for blog post parsing
- Multiple 10+ minute cycles for simple changes
- High computational cost vs. human engineer efficiency
- No persistence of learned knowledge between tasks
Philosophical Concerns
- "In the future, is everything mid?" - Ryan Carniato's key question
- AI may propagate average expectations rather than drive innovation
- Popularity-driven decisions rather than optimal technical choices
- Risk of losing human expertise and technical decision-making capability
Developer Job Impact Assessment
- Speaker not worried about job displacement
- Writing code is "easiest part" of development job
- Other responsibilities (architecture, debugging, maintenance) much harder
- Elon Musk Twitter layoffs had more impact on dev jobs than current AI tools
- Tools like this may help with scaffolding but won't replace skilled engineers
Final Assessment
- Extremely skeptical of Devon's capabilities and marketing claims
- Poor presentation quality undermines credibility
- Technology appears to be basic LLM wrapper rather than breakthrough
- Useful for simple scaffolding but far from replacing human engineers
- Recommends learning to code rather than waiting for AI tools
- "If your idea isn't good enough to motivate learning to code, it isn't good enough to build"
