AI Engineer World's Fair 2025
Does AI Actually Boost Developer Productivity? (Stanford / 100k Devs Study)
Loading slides…
Read the talk
Does AI Actually Boost Developer Productivity? (100k Devs Study)
Yegor Denisov-Blanch presents evidence from enterprise Git histories showing that AI coding tools increase output on average, but rework, task complexity, project maturity, language popularity, and codebase size determine how much becomes useful progress.
From a talk by Yegor Denisov-Blanch
At a glance
Ideas worth remembering
Count delivered functionality rather than commits, pull requests, tasks, or lines of code; AI-generated repairs can inflate every activity metric without creating equivalent progress.
The talk’s aggregate estimate falls from a 30–40% apparent output increase to roughly 15–20% net productivity after rework and bug fixing.
Low-complexity Greenfield work shows the strongest gains; high-complexity Brownfield work shows indicative gains of only 0–10% and can sometimes become slower.
Widely used languages benefit more consistently than the lower-popularity examples in the talk, where unreliable suggestions can erase gains on complex work.
Treat the codebase-size relationship as a hypothesis rather than a settled result: the talk labels it illustrative and proposes effective-context limits, noise, dependencies, and domain logic as possible causes.
Replace the replacement story with a measurement question
The talk opens with executive pressure to treat AI adoption as a race toward replacing developers. Denisov-Blanch offers a more measured starting point: AI coding assistance can increase developer productivity, but it can also decrease it. The useful question is whether generated code helps a particular team deliver useful functionality faster.
The research group approaches that question with longitudinal and cross-sectional data. Historical Git records let it examine changes over time, including work recorded before a company joined the study. Comparisons across more than 600 enterprises, midsized companies, and startups provide the second dimension. The dataset described in the talk contains more than 100,000 software engineers, dozens of millions of commits, and billions of lines of code.
Most observations come from private repositories. Denisov-Blanch argues that this makes team-level measurement more self-contained than public-repository activity, which may represent occasional or weekend work. That choice improves coverage of organizational coding activity, but the method still centers on changes visible in Git; the talk does not claim that repository history captures every valuable activity performed by an engineer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
More activity can conceal less useful progress
The first measurement trap is counting commits, pull requests, completed tasks, or shorter intervals between commits. Tasks differ in size, so a higher count does not establish that a team delivered more functionality. AI compounds the problem when generated code creates bugs and therefore creates additional repair tasks. Both the original change and its fix increase activity, even if the team has mostly spun its wheels.
The second trap is generalizing from clean-slate experiments. When one group receives AI assistance and another does not, the assigned work is often a Greenfield task with little prior context. Models are especially effective at producing boilerplate in that setting. Established software work is usually Brownfield: engineers must understand existing code, dependencies, conventions, and domain logic. A result from an isolated new project does not automatically transfer to that environment.
The third trap is self-reporting. In a small experiment with 43 developers, participants placed themselves in five-percentile buckets relative to a global mean or median. Their estimates missed measured productivity by about 30 percentile points, and only one in three placed themselves within the correct quartile. Surveys can still reveal morale, satisfaction, and other experiences that code metrics cannot see. Denisov-Blanch’s narrower judgment is that they should not serve as the productivity measure itself.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Measure delivered functionality, then account for rework
The proposed method begins with an expensive ideal: ask a panel of 10 to 15 independent engineering experts to evaluate a code change for qualities such as maintainability, output, quality, and the effort required to produce it. Denisov-Blanch says these panels agree with one another and that their aggregate assessments predict real-world outcomes. Repeating that process for every commit, however, would be too slow and costly.
The group therefore built a model intended to automate the panel’s assessment. It connects to Git, examines the source-code changes in each commit, and scores the change across similar dimensions. Authors, SHAs, and timestamps locate those assessments by person and time. In this framework, team productivity is the functionality delivered over time—not the number of commits or lines written. The model is an approximation that the speaker says correlates well with expert review; the talk does not provide the correlation statistic or a full validation protocol.
A pilot involving about 120 developers shows why the distinction matters. Monthly output is divided into added functionality, removed functionality, refactoring, and rework. Refactoring changes older code and may or may not be wasteful. Rework changes much more recent code, so the framework treats it as corrective work that reduces the value of the initial output. After AI entered the team’s workflow, both apparent production and rework rose.
The headline arithmetic is simple but consequential. AI-assisted coding appears to raise output by roughly 30–40%, yet bug fixes and other rework consume a substantial part of that increase. The aggregate net improvement presented in the talk is about 15–20% across industries and sectors. That average is useful as a baseline, not a forecast for every team—the distribution matters more than the mean.
From generated output to net productivity
Read the diagram as text
- AI-assisted coding. Developers produce more apparent output.
- 30–40% apparent gain. The initial estimate reflects more code and functionality being delivered.
- Bugs and recent-code rework. Developers revisit recent generated changes and repair problems.
- 15–20% average net gain. Corrective work reduces the aggregate improvement.
- AI-assisted coding → 30–40% apparent gain: increases production.
- 30–40% apparent gain → Bugs and recent-code rework: some output needs repair.
- Bugs and recent-code rework → 15–20% average net gain: subtract correction cost.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Complexity and project maturity move the result
The productivity distributions begin as low as −20%, which immediately rules out a universal positive effect. Simpler work performs better than complex work, and low-complexity Greenfield tasks show the strongest distribution of gains. Denisov-Blanch limits this comparison to enterprise settings; he expects personal Greenfield projects to behave differently and potentially show larger improvements.
High-complexity tasks produce smaller gains and sometimes make engineers slower. Brownfield work also benefits less than Greenfield work. The charts summarize both a central estimate and an interquartile range, with the line spanning roughly the 25th to 75th percentiles. This matters because two teams doing nominally similar work can still land on different sides of the average. Denisov-Blanch also says the causes of the negative cases are not yet clear.
A simplified matrix based on 136 teams across 27 companies makes the interaction concrete: low-complexity Greenfield work shows indicative gains of 30–40%; high-complexity Greenfield work, 10–15%; low-complexity Brownfield work, 15–20%; and high-complexity Brownfield work, 0–10%. The speaker describes these as orientative guidelines. They can help decide where to pilot AI, but they are not precise promises for a project plan.
Language popularity introduces another source of variance. For widely used languages such as Python, Java, JavaScript, and TypeScript, the talk reports gains around 20% on simpler work and 10–15% on complex work. Assistance is less dependable for the lower-popularity examples given—COBOL, Haskell, and Elixir—and complex tasks in those languages can become slower. A tool that helps only two times out of five may be abandoned even if its occasional suggestions are good.
Indicative gains by task and project type
Read the diagram as text
- Greenfield · low complexity. Greenfield — 30–40% indicative gain
- Greenfield · high complexity. Greenfield — 10–15% indicative gain
- Brownfield · low complexity. Brownfield — 15–20% indicative gain
- Brownfield · high complexity. Brownfield — 0–10% indicative gain
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A large context window does not equal effective codebase understanding
The final analysis is explicitly more theoretical and less empirically established. Denisov-Blanch presents an illustrative curve in which AI productivity gains fall sharply as codebases grow from 1,000 to 10 million lines. He proposes three mechanisms: finite effective context, a worsening signal-to-noise ratio, and the growing number of dependencies and domain-specific rules in larger systems.
An advertised context-window capacity does not show that a model will use every token equally well. Citing a paper comparison, the talk describes coding-task performance declining as context increases from 1,000 to 32,000 tokens; one illustrated model falls from roughly 90% to 50% by 32,000 tokens despite supporting a much larger maximum window. The extrapolation to 64,000 or 128,000 tokens is rhetorical, not a measured result presented here.
The practical conclusion is selective adoption. Use AI in many cases, but evaluate it against the work actually being done. Teams should segment results by task complexity, Greenfield versus Brownfield status, language popularity, codebase size, and supplied context. Most importantly, they should measure useful functionality and the rework needed to preserve it. A universal license rollout may be easy; a universal productivity forecast is not.
Denisov-Blanch closes by directing interested readers to the Stanford Software Engineering Productivity Research portal. The official AI Engineer talk page provides the recording, chapters, transcript, and a concise reading version.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The research portal named at the end of the talk, with information about the group’s engineering-productivity research.
The official AI Engineer page for the recording, with chapters, highlights, transcript, and a reading version.
Further reading
- Yegor Denisov-BlanchReference
The speaker’s site describes his developer-productivity research, publications, and subsequent work on measuring AI-assisted engineering.
A later presentation extending the measurement approach with matched teams, usage data, codebase cleanliness, guardrail metrics, and an enterprise case study.
Related talks
- How to Quantify AI ROI in Software Engineering (Stanford Study / 120k Devs)
Denisov-Blanch extends this framework into matched-team analysis, usage measurement, guardrail metrics, and a detailed enterprise adoption case.
- No Vibes Allowed: Solving Hard Problems in Complex Codebases
Dex Horthy examines why coding agents create rework in established codebases and presents context engineering as a practical response to the Brownfield difficulty described here.
- Why Agent Hype can fall short of reality – Joel Becker, METR
Joel Becker contrasts benchmark gains with a randomized study of experienced developers working on messy, context-dependent software tasks.
Full transcript
Loading transcript…