AI summary by TwinMind
Cracking the Code on Offensive Security With AI ft XBOW CEO and GitHub Copilot Creator Oege de Moor
Sequoia Capital
51m 37s
AI Security Threat Landscape
- AI code generation enables anyone to create code without security knowledge
- Foundation models trained on public source code containing vulnerabilities
- Result: more code generated with more security problems
- Attackers already using AI to enhance their effectiveness
- Creates greater threat: more code + more attacks = need for automation like Expo
Expo Company Introduction
- Founded by Oiad Deore, creator of GitHub Copilot
- Former Oxford computer science professor
- Previous startup Seml acquired by GitHub
- Expo: AI penetration testing company automating offensive security
- Example of "Service as Software" model
- 13 people at launch of results
Benchmark Performance Results
- Initial industry benchmarks from commercial providers (Ports Wigger, Pentland): 75% success rate
- Suspected training data contamination, created original benchmarks guaranteed not in training sets
- Original benchmarks scored even better: 85% success rate
- Comparison with 5 professional pentesters from reputed firms on same 24 challenges
- Top-tier pentester (multi-billion dollar hedge fund level) scored same as AI
- Human took 40 hours vs AI took 28 minutes
- Results exceeded expectations - thought would take at least 1 year to reach reasonable proficiency
Real-World Vulnerability Discovery
- Working with early design partners daily
- Found critical vulnerability at major financial institution this morning
- AI process:
- Crawled web app, found PHP source code
- Discovered insecure signing algorithm for host access
- Continued crawling, found another endpoint
- Reused previous technique with modifications
- Found JavaScript code with candidate parameters
- Gained access to endpoint intended for PDF downloads
- Discovered could download password files
- Demonstrates creative, multi-step reasoning similar to human experts
- "Something like that every day"
Offensive Security Market Definition
- Current best practice: invite external experts to simulate attacks
- Highly skilled activity requiring years of training
- Expensive and slow: typical penetration test costs $18,000
- Only performed 1-2 times per year due to cost
- Systems evolve faster than testing frequency
- Expo automates this process for continuous testing instead of periodic
Market Transformation Through AI
- Current market small due to limited highly skilled human experts
- AI expanding market fundamentally:
- More code generation → more vulnerabilities
- Attackers using AI → greater threats
- Automation becomes essential
- Market expected to grow enormously
- Analogy to military war games and red teaming exercises
- National security implications - "first ever AI cyber warrior"
Technology Architecture
- Most magic built on top of foundation models, not within them
- Work with multiple foundation model providers
- Benefit from competition between providers
- Key components:
- Benchmarks organized as curriculum to teach models
- Specialized tools for LLMs (custom web browser for form filling/clicking)
- Guard rails to prevent harmful actions (prevent "drop table" type disasters)
- Attack surface discovery to prioritize targets
- Prompt engineering and chain-of-thought prompting
Technical Challenges and Solutions
- Hallucination management: run multiple attempts, better models need fewer attempts
- Stochastic nature handled through repetition
- Automated validator ensures reports are correct and reproducible
- Human review still required for final fixes
- Described AI as "overeager super brilliant teenager" needing supervision
Data and Training Approach
- Benchmarks critical for evaluation and improvement
- Multiple data sources:
- Security expert contractors creating new benchmarks
- Mining open source (Docker Hub images)
- Self-generated data from successful attacks
- Reinforcement learning on specific examples
- Similar improvement trajectory expected as other games like Go
Cloud-Only Deployment Strategy
- Available only in cloud, not downloadable software
- Enables control over scope and usage
- Customers must prove legitimate ownership of targets
- Prevents misuse for attacking others' systems
- Security measure against potential bad actors
Pricing Model Innovation
- Hybrid approach combining subscription and consumption:
- Base license based on engineering team size (similar to GitHub)
- Additional "attack hours" for thorough testing
- Fixed attack hours included with base license
- Extra service fees for deeper testing
- Clear correlation between resources consumed and customer benefit
- Enables continuous testing after every code change
Future Roadmap
- Currently in careful early design partner phase with human supervision
- Once supervision no longer needed, expect rapid deployment
- Easy deployment: point at existing servers for immediate results
- Goal: significantly transform web security by next summer
- Plans for open source demonstrations and platforms like HackerOne
Broader AI Impact Predictions
- Short term: customer support disruption already happening
- Social media problems could be mitigated with AI
- Long term: health and biology biggest impact areas
- Coding transformation: more people able to create software
- Professionals focus on conceptual ideas and architecture
- Future: conversation with model becomes the artifact, not code
- "English is the coding language"
Personal Journey and Philosophy
- Left Oxford professorship for commercial impact and speed
- Small teams with profit motive move faster than academic settings
- GitHub Copilot launched with only 10 people
- Focus and speed as key startup advice
- Imperative to build this technology before "bad guys" do
- Protecting the Free World's software infrastructure
