Ian Butler is an AI researcher at XBOW focused on autonomous offensive security and the cofounder and former chief executive of Bismuth, where he helped develop the SM-100 benchmark for evaluating whether coding agents can identify and repair real software defects. His research confronts a practical limitation of automated programming: generating plausible code does not guarantee that an agent can understand an existing system, recognize genuine vulnerabilities, or verify its own repairs.
Butler met future cofounder Nick Gregory through New York University’s OSIRIS cybersecurity laboratory. In 2019, they began building a technical-documentation search product for developers’ editors and internal knowledge bases. His writing on web crawlers, search relevance, and private search reflects an early interest in retrieving useful information from complex collections.
Before founding Bismuth, Butler worked as a senior engineer on Zillow’s experimentation platform. At Bismuth, he and Gregory built software that transformed development tickets into pull requests, reviewed code, identified vulnerabilities, and tested proposed fixes. The company received backing from Aviso Ventures in March 2025.
Their SM-100 benchmark assembled 100 previously repaired bugs from more than 84 public repositories to test agents on defect discovery, false-positive rates, pull-request review, and remediation. After winding down Bismuth, Butler and Gregory joined XBOW as AI researchers, applying their experience with automated bug detection to offensive security.
What reliable coding agents require
- Software maintenance demands architectural understanding. Finding defects requires tracing data flows, understanding calling context, and distinguishing genuinely dangerous behavior from code that only appears suspicious in isolation.
- False positives create developer alert fatigue. Agents that produce large volumes of incorrect warnings make real problems harder to identify. Butler evaluates success by whether engineers receive useful, trustworthy findings without repeatedly rerunning the same analysis.
- Cross-file reasoning depends on preserved context. Summarizing away essential files can erase relationships needed to diagnose complex bugs. Butler recommends focused diffs, inventories of classes and variables, and explicit mappings between components; reasoning-oriented models improve detection but still discover different defects across repeated runs.
- Security guidance and repairs need verification. He recommends grounding prompts in the OWASP Top Ten, specifying vulnerabilities such as SQL injection and authentication bypass, and requiring test-verified remediation before changes enter a codebase.
Butler also develops Chevalier, an agent-runtime toolkit spanning Rust, TypeScript, and Python, and helps organize AI Tinkerers in San Francisco.
