AI summary by TwinMind

Factory’s Matan Grinberg and Eno Reyes Unleash the Droids on Software Development | Training Data

Sequoia Capital

59m 11s

Industry Perception and Motivation

  • "Agent" term still associated with unreliable, stochastic, demo-ware, and vaporware after 13 months
  • Factory's mission: build systems valuable today for enterprise engineers, not just hackers or demos

Founder Backgrounds and Meeting

  • Matan Grimberg:
    • Studied string theory at Princeton, cold-called Juan Maldacena (famed physicist) for advising
      • Maldacena's 2-hour chalkboard session, challenge to solve a problem within 24 hours
      • Published paper together
    • Later at Berkeley (PhD, theoretical physics), realized pursuit due to challenge, not passion
    • Saw three options: quantitative finance, big tech, startups
    • Shifted research to ML and AI
    • Cold-emailed Sean Maguire (former string theorist, now at Sequoia) after seeing him speak
      • 3-hour walk led to advice: drop out, join Twitter, join Foundry, or start a company
      • Reconnected with Eno at a LangChain hackathon, daily collaboration since
  • Eno Rees:
    • Brown CS graduate, ML engineer at Hugging Face, also at Microsoft
    • Family's immigrant story, drive from parents
    • Early work on LLMs for enterprise, LangChain integrations
  • Founders met in undergrad, mutual friends, only connected after moving to Bay Area

Factory's Strategic Approach

  • Decided against training foundation models; focused on application layer
  • Rationale:
    • Software is the lever on the world; AI code generation is a lever on software (compound lever)
    • Code is fundamental to machine intelligence
    • AI application layer offers the most opportunity for impactful products

Factory Product and Technical Decisions

  • Factory: automating software development lifecycle tasks (code review, documentation, testing, debugging, refactoring)
    • Focus on tasks engineers dislike, meeting enterprise needs
    • Droids: autonomous systems, job-specific agents (e.g., review Droid, test Droid)
      • Each Droid mapped to a task in the SDLC, framed as "games"
    • Name "Droid" chosen to avoid "agent" stigma; originally incorporated as San Francisco Droid Company, changed due to Lucasfilm trademarks
  • Technical philosophy:
    • Cognitive architectures: inspired by DeepMind's "game-like" reasoning (e.g., AlphaZero for Go, protein folding)
    • Most reasoning at inference time—planning, search, feedback loops, not training new foundation models
    • Application layer is locus for innovation (latency, context windows, orchestration)
    • Not a single cognitive architecture for all tasks; choose per-task design (e.g., Monte Carlo tree search, language agent tree search)

Metrics, Benchmarks, and Real-World Impact

  • Focus on system-wide org metrics over individual developer productivity
    • Key metrics: engineering time saved, speed, code quality, engineering velocity
    • Avoid narrow metrics like autocomplete acceptance; instead, track org-wide code churn, open-to-merge time, code cycle times
    • Copilot increases code churn by 20-40% in some studies; Factory aims to reduce churn and increase velocity
  • Customer results:
    • Average cycle time improvement: 22%
    • Code churn reduction: 133%
    • Test Droid saves ~40 minutes/engineer/day
  • Thousands of users provide live benchmarks, rapid iteration
  • Example: Factory's own Droids built GitLab integration autonomously, replacing intern-level work

Human/AI Interaction and Adoption

  • Engineers value ownership; tools must be flexible to developer preferences
    • Code review is DMV-like: universally disliked but necessary
    • Test Droid is loved by developers, less obviously linked to business metrics for leaders
  • Adoption already happening for tasks like code review, testing (intern-level automation)
  • Higher-level tasks (e.g., monorepo refactors) will be gradually decomposed into sub-tasks handled by AI; humans shift to orchestration/architecture roles

Competitive Landscape and Execution

  • Factory's competitive edge: team more "obsessed" than others, executes faster
  • Only way to win: execute rapidly, stay mission-driven, ignore noise from competitors with same access to models

Philosophy and Vision

  • Factory aligns with developers by automating tasks they dislike, not the creative/ownership parts
  • In 5 years, "software engineering" role may transform—developers become curators/orchestrators, not just coders
  • Goal: enable creative freedom, not replacement
  • Advice to founders: "Go for gold," build products that get 10x better with new model releases, don't fear foundation model advances

S-Bench and Technical Achievements

  • S-Bench: 2,300 real-world issues from 12 open-source Python projects; tests AI's ability to fix actual bugs/tasks
    • Previous SoTA: 14% (Cognition Dev), Factory achieved 19%
    • Outperformed tech giants (Microsoft, Amazon, IBM, Bytedance)
  • Approach: droids built for real customers, not benchmarks; cognitive architectures mapped to real human workflows
    • Emphasis on planning, task decomposition, environmental grounding, codebase understanding
  • Each Droid has its own cognitive architecture, balancing workflow rigidity and flexibility
  • Factory's own internal tasks (e.g., GitLab integration) completed autonomously by Droids

Benchmarking, Reliability, and Future Milestones

  • S-Bench not sole adoption metric—real-world reliability already higher for customer tasks
  • AI interaction design: systems must accommodate failure, allow human oversight, interpretability
  • Predicts S-Bench will evolve (S-Bench 2/3) before 80-90% solved; expects 2-3 years for major progress
  • Collaboration with Princeton team (creators of S-Bench)

AI Industry Admiration

  • Jeff Dean (Google/DeepMind): respected for research, leadership, responsible approach
  • Collaboration over "solo genius" emphasized; admires Mistral's speed and mission in open-source
  • If not working on Factory, Eno would work on robotics; Matan can't imagine working on anything else

Closing Advice

  • Build for compounding improvements from foundation models, not competition with them
  • "Land of picks and shovels, no one's struck gold yet—go for gold"
  • Stay excited about new models; if worried by new releases, reconsider product strategy

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome