AI summary by TwinMind

A Measured Take on Devin

Theo - t3․gg

44m 41s

Initial Reaction to Devon AI Announcement

  • Cognition Labs posted video on Twitter causing massive Dev World buzz
  • Half of speaker's feed/DMs about Devon AI software engineer
  • Devon claims: first AI software engineer, state-of-the-art on SWE-Bench coding benchmark
  • Passed practical engineering interviews from leading AI companies
  • Completed real jobs on Upwork
  • Speaker notes engineers exist who meet these criteria but still perform poorly

SWE-Bench Performance Analysis

  • Devon resolves 13-14% of GitHub issues without assistance on SWE-Bench
  • Previous models: 1.9-2% unassisted, up to 5% assisted
  • Tool covering only 14% of issues "isn't as good as an engineer"
  • Speaker remains curious but not impressed by initial claims

Demo Video Breakdown - API Benchmarking

  • Devon asked to benchmark llama performance on different API providers
  • Creates step-by-step plan, uses command line, code editor, browser
  • AI opens documentation to learn APIs autonomously
  • Chat interface shows human providing guidance throughout (hyper-speed typing noted)
  • Human inputs: "measure wall clock run time of API," "use REST API directly," "build whole project using only metric we care about is tokens per second"
  • Devon encounters unexpected error, adds debugging print statement, reruns code
  • Builds and deploys website with full styling as visualization
  • Speaker criticizes poor video editing quality affecting retention

Game of Life Implementation Analysis

  • Devon deployed functional Game of Life on Netlify
  • Console errors present: "prevent defaults not a function of e-mouse drag"
  • Time analysis: 1:40-1:41 initial creation, 1:41-1:51 production build (10 minutes total)
  • Subsequent changes: 7+ minutes per modification
  • Bug fix: 3 minutes (1:53-2:11)
  • Frame rate changes: 4 minutes
  • Responsive design + interactivity: additional time
  • Speaker notes 10+ minutes per change is problematic for iteration

Blog Post Learning Demo

  • Sarah demonstrates Devon learning from blog post to generate desktop background images
  • Time stamps: 12:59 start to 1:41 completion = 42 minutes
  • Generated one good image (Sarah) and two difficult-to-read bonus images
  • Speaker notes: similar time to decent engineer but without retained knowledge
  • AI must retrain for each similar task vs. human learning persistence

Bug Fix Demo - Senpai Repository

  • Devon fixes log function bug in Python algebra system
  • Total time: 1:16-12:12 = 1 hour
  • Breakdown: 12 minutes setup/bug identification, 30-40 minutes debugging/fixing, 30 minutes testing/cleanup
  • Issue: integer division causing log(0) = infinity
  • Fixed by replacing integer division with true division
  • Speaker questions if test suite was slow or if AI process was inefficient

Audio Quality Criticism

  • Multiple demos have poor audio quality
  • Speaker demonstrates Adobe audio enhancement tool
  • Criticizes lack of production quality for 500K+ impression videos
  • Suggests using AI for presentation improvement rather than code generation

Skepticism Points

  • No live typing demonstrations - all pre-recorded scrolling through completed work
  • Cognition hiding behind waitlist rather than public access
  • Comparison to OpenAI's Sora transparency about limitations
  • OpenAI shows failure cases and current weaknesses openly

Technical Architecture Analysis

  • Bloomberg article reveals: combines LLMs (GPT-4) with reinforcement learning
  • Not building own models, just wrapping GPT-4
  • 3 million token context claimed (vs GPT-4.5 rumored 256K = 12x difference)
  • Speaker notes this approach easily replicable by competitors in 3-6 months
  • Company raised $21 million from Peter Thiel's Founders Fund
  • 10-person team, incorporated only 2 months ago

Founder Background

  • Woo Brothers: world-renowned competitive programmers since teenagers
  • Helped elevate US national coding team against Chinese/Eastern European rivals
  • One brother (Yan) at Harvard, hasn't told parents about dropping out
  • Previous experience at Scale AI (noted as "more early than talented")
  • CEO demonstrated exceptional math competition skills as child

Real-World Testing Results

  • Matt Bielman (Netlify CEO) built todo app with Devon
  • App has multiple bugs: add task button doesn't work, no local storage, dark mode toggle breaks
  • Ryan Carniato (Solid JS creator) analysis: 130KB JavaScript bundle, ~80 Lighthouse score
  • Uses React + Chakra UI + Framer Motion for simple todo app
  • 26x larger than equivalent implementations in Svelte/Preact/Solid (5KB vs 130KB)
  • Described as "confidently mid" - replicating intern-level work

GitHub Analysis

  • Devon has GitHub account with rust code contributions
  • Code quality issues: awful comments, default exit codes for errors
  • Accidentally includes test data in production changes
  • Represents realistic developer mistakes but not professional quality

Security Concerns

  • Generated code contains security vulnerabilities
  • Uses insecure authentication patterns that bypass layout checks
  • Doesn't use modern React patterns (React Query, proper state management)
  • Race conditions present in async operations

Comparison to Competitors

  • Magic AI raised $100M, building own models (more legitimate approach)
  • Marius (Y Combinator company) - honest about limitations, publicly available
  • Marius focuses on scaffolding rather than claiming to replace engineers
  • Speaker invested in Marius due to founder transparency and realistic scope

Cost and Efficiency Analysis

  • 40+ minutes of GPU-heavy computation for blog post parsing
  • Multiple 10+ minute cycles for simple changes
  • High computational cost vs. human engineer efficiency
  • No persistence of learned knowledge between tasks

Philosophical Concerns

  • "In the future, is everything mid?" - Ryan Carniato's key question
  • AI may propagate average expectations rather than drive innovation
  • Popularity-driven decisions rather than optimal technical choices
  • Risk of losing human expertise and technical decision-making capability

Developer Job Impact Assessment

  • Speaker not worried about job displacement
  • Writing code is "easiest part" of development job
  • Other responsibilities (architecture, debugging, maintenance) much harder
  • Elon Musk Twitter layoffs had more impact on dev jobs than current AI tools
  • Tools like this may help with scaffolding but won't replace skilled engineers

Final Assessment

  • Extremely skeptical of Devon's capabilities and marketing claims
  • Poor presentation quality undermines credibility
  • Technology appears to be basic LLM wrapper rather than breakthrough
  • Useful for simple scaffolding but far from replacing human engineers
  • Recommends learning to code rather than waiting for AI tools
  • "If your idea isn't good enough to motivate learning to code, it isn't good enough to build"

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome