AI summary by TwinMind
Factory’s Matan Grinberg and Eno Reyes Unleash the Droids on Software Development | Training Data
Sequoia Capital
59m 11s
Industry Perception and Motivation
- "Agent" term still associated with unreliable, stochastic, demo-ware, and vaporware after 13 months
- Factory's mission: build systems valuable today for enterprise engineers, not just hackers or demos
Founder Backgrounds and Meeting
- Matan Grimberg:
- Studied string theory at Princeton, cold-called Juan Maldacena (famed physicist) for advising
- Maldacena's 2-hour chalkboard session, challenge to solve a problem within 24 hours
- Published paper together
- Later at Berkeley (PhD, theoretical physics), realized pursuit due to challenge, not passion
- Saw three options: quantitative finance, big tech, startups
- Shifted research to ML and AI
- Cold-emailed Sean Maguire (former string theorist, now at Sequoia) after seeing him speak
- 3-hour walk led to advice: drop out, join Twitter, join Foundry, or start a company
- Reconnected with Eno at a LangChain hackathon, daily collaboration since
- Studied string theory at Princeton, cold-called Juan Maldacena (famed physicist) for advising
- Eno Rees:
- Brown CS graduate, ML engineer at Hugging Face, also at Microsoft
- Family's immigrant story, drive from parents
- Early work on LLMs for enterprise, LangChain integrations
- Founders met in undergrad, mutual friends, only connected after moving to Bay Area
Factory's Strategic Approach
- Decided against training foundation models; focused on application layer
- Rationale:
- Software is the lever on the world; AI code generation is a lever on software (compound lever)
- Code is fundamental to machine intelligence
- AI application layer offers the most opportunity for impactful products
Factory Product and Technical Decisions
- Factory: automating software development lifecycle tasks (code review, documentation, testing, debugging, refactoring)
- Focus on tasks engineers dislike, meeting enterprise needs
- Droids: autonomous systems, job-specific agents (e.g., review Droid, test Droid)
- Each Droid mapped to a task in the SDLC, framed as "games"
- Name "Droid" chosen to avoid "agent" stigma; originally incorporated as San Francisco Droid Company, changed due to Lucasfilm trademarks
- Technical philosophy:
- Cognitive architectures: inspired by DeepMind's "game-like" reasoning (e.g., AlphaZero for Go, protein folding)
- Most reasoning at inference time—planning, search, feedback loops, not training new foundation models
- Application layer is locus for innovation (latency, context windows, orchestration)
- Not a single cognitive architecture for all tasks; choose per-task design (e.g., Monte Carlo tree search, language agent tree search)
Metrics, Benchmarks, and Real-World Impact
- Focus on system-wide org metrics over individual developer productivity
- Key metrics: engineering time saved, speed, code quality, engineering velocity
- Avoid narrow metrics like autocomplete acceptance; instead, track org-wide code churn, open-to-merge time, code cycle times
- Copilot increases code churn by 20-40% in some studies; Factory aims to reduce churn and increase velocity
- Customer results:
- Average cycle time improvement: 22%
- Code churn reduction: 133%
- Test Droid saves ~40 minutes/engineer/day
- Thousands of users provide live benchmarks, rapid iteration
- Example: Factory's own Droids built GitLab integration autonomously, replacing intern-level work
Human/AI Interaction and Adoption
- Engineers value ownership; tools must be flexible to developer preferences
- Code review is DMV-like: universally disliked but necessary
- Test Droid is loved by developers, less obviously linked to business metrics for leaders
- Adoption already happening for tasks like code review, testing (intern-level automation)
- Higher-level tasks (e.g., monorepo refactors) will be gradually decomposed into sub-tasks handled by AI; humans shift to orchestration/architecture roles
Competitive Landscape and Execution
- Factory's competitive edge: team more "obsessed" than others, executes faster
- Only way to win: execute rapidly, stay mission-driven, ignore noise from competitors with same access to models
Philosophy and Vision
- Factory aligns with developers by automating tasks they dislike, not the creative/ownership parts
- In 5 years, "software engineering" role may transform—developers become curators/orchestrators, not just coders
- Goal: enable creative freedom, not replacement
- Advice to founders: "Go for gold," build products that get 10x better with new model releases, don't fear foundation model advances
S-Bench and Technical Achievements
- S-Bench: 2,300 real-world issues from 12 open-source Python projects; tests AI's ability to fix actual bugs/tasks
- Previous SoTA: 14% (Cognition Dev), Factory achieved 19%
- Outperformed tech giants (Microsoft, Amazon, IBM, Bytedance)
- Approach: droids built for real customers, not benchmarks; cognitive architectures mapped to real human workflows
- Emphasis on planning, task decomposition, environmental grounding, codebase understanding
- Each Droid has its own cognitive architecture, balancing workflow rigidity and flexibility
- Factory's own internal tasks (e.g., GitLab integration) completed autonomously by Droids
Benchmarking, Reliability, and Future Milestones
- S-Bench not sole adoption metric—real-world reliability already higher for customer tasks
- AI interaction design: systems must accommodate failure, allow human oversight, interpretability
- Predicts S-Bench will evolve (S-Bench 2/3) before 80-90% solved; expects 2-3 years for major progress
- Collaboration with Princeton team (creators of S-Bench)
AI Industry Admiration
- Jeff Dean (Google/DeepMind): respected for research, leadership, responsible approach
- Collaboration over "solo genius" emphasized; admires Mistral's speed and mission in open-source
- If not working on Factory, Eno would work on robotics; Matan can't imagine working on anything else
Closing Advice
- Build for compounding improvements from foundation models, not competition with them
- "Land of picks and shovels, no one's struck gold yet—go for gold"
- Stay excited about new models; if worried by new releases, reconsider product strategy
