James Shi is a founding engineer at Datacurve, building the infrastructure, expert-generated training data, and evaluations behind advanced coding models. His signature project, DeepSWE, measures whether coding agents can solve unfamiliar, demanding software-engineering problems without exploiting leaked answers or brittle grading.
Shi studied computer science and business at the University of British Columbia before leaving school to build UncleGPT, an early product aimed at product managers. The venture evolved into Datacurve, a Y Combinator-backed coding-data company founded by Serena Ge and Charley Lee. As a founding engineer, Shi has helped scale technical infrastructure, data pipelines, expert review, and quality control.
The open-source DeepSWE benchmark contains 113 original tasks across 91 repositories and five languages: TypeScript, JavaScript, Python, Rust, and Go. Its assignments approximate realistic, open-ended development work while addressing problems that undermine evaluations derived from public pull requests.
How Shi evaluates coding agents
- Contamination-resistant coding benchmarks: Public pull requests can expose solutions through training data, online discussions, and Git history. DeepSWE uses newly authored tasks; version 1.1 further isolates agent and verifier runtimes and removes unnecessary Git references and commit history.
- Behavior-oriented verification: Correct solutions should pass because they satisfy the requested behavior, regardless of private helper names or implementation choices. DeepSWE’s published research describes functional verifiers designed to reduce both false positives and false negatives.
- Realistic long-horizon engineering tasks: Concise, high-level prompts require agents to explore unfamiliar repositories, follow existing conventions, and coordinate changes across multiple files. Shi identifies debugging, refactoring, and greater repository diversity as important areas for future benchmarks.
- Model capability versus agent scaffolding: A shared evaluation harness helps distinguish underlying model performance from surrounding prompts, tools, and workflows. Shi extends this argument to practical AI application design, emphasizing retrieval, orchestration, memory, and tool integration.
- Self-verification as an engineering behavior: Shi examines whether agents test their own patches, preserve multipart requirements, and respect repository interfaces. His AI Engineer conference presentation demonstrates how even a brief instruction implying that tests are handled can discourage otherwise capable models from checking their work.
