The Productivity Data on AI Coding Tools Doesn’t Agree. That’s the Interesting Part.

In July 2025, the nonprofit research organisation METR ran a randomised controlled trial, the same method used in clinical drug trials, to measure how AI tools actually affect developer productivity. Sixteen experienced open source developers completed 246 real tasks in mature repositories they knew well, with each task randomly assigned to allow or disallow AI tools. Before starting, the developers forecast that AI would cut their completion time by 24%.

The measured result was the opposite. Developers took 19% longer when AI tools were allowed. Afterwards, even having lived through the slower experience themselves, they still estimated that AI had sped them up by about 20%.

It isn’t the only data point, and it doesn’t point the same way as some others

A different randomised trial, run inside a large enterprise with developers working on assigned tasks rather than their own open source projects, found AI use made developers roughly 21% faster, with more senior and more frequent coders seeing the largest gains. Google has reported similar internal figures from controlled testing with its own engineers. GitHub’s own research on Copilot has reported task completion speedups as high as 55.8% in some conditions.

So the honest picture is not “AI slows developers down.” It’s that different studies, in different settings, are finding meaningfully different answers, and the disagreement itself is worth paying attention to rather than picking whichever result fits the narrative you already believed.

What might explain the gap

One plausible difference is what kind of work was being measured. METR’s participants were maintainers working in codebases they already knew intimately, often with years of accumulated context in their own heads. For that kind of work, an AI tool has to earn back the time spent describing context, checking suggestions against tacit knowledge only the maintainer has, and correcting output that looks right but doesn’t fit the codebase’s specific history. Enterprise studies more often measure developers on assigned, less personally familiar tasks, where an AI tool’s ability to produce a working first draft quickly has more room to help rather than compete with existing expertise.

There’s a second data point worth bringing in here, because it points at the same mechanism from a different angle. A large-scale empirical study of AI-generated code in real repositories found that the volume of unresolved, AI-introduced technical debt, tracked as issues still present at the latest snapshot of each repository, climbed from a few hundred in early 2025 to more than 100,000 by February 2026. Separately, researchers studying AI-generated pull requests found that AI coding agents produce meaningfully more redundant, duplicated code than human developers do, but that human reviewers don’t rate those pull requests any more critically. The debt accumulates quietly because the code looks fine on the surface.

Put together, these studies suggest a possible reconciliation: AI tools can produce a working answer fast, which is what a single-task productivity trial captures. Whether that speed holds up depends on how much unreviewed debt that fast answer quietly deposits into the codebase, which a single-task trial generally doesn’t measure at all.

METR’s own response to this is worth noting

To its credit, METR didn’t treat its own surprising result as final. The organisation has since flagged that the original study may have had selection effects worth correcting for, and started an expanded follow-up with a larger, more diverse set of developers. That’s a more honest posture than either side of this debate usually takes in public: publish a striking result, and then go check whether the result actually holds up under a better design, rather than either defending it forever or quietly moving on.

What this means for how you think about hiring

None of this settles whether AI tools make developers faster. It’s genuinely unresolved, and anyone claiming otherwise, in either direction, is overstating what the current research supports. What does seem more consistent across the studies is the code quality question: AI-assisted code tends to accumulate more unreviewed debt unless someone is actively catching it. That’s a more useful thing to hire for than a claim about raw speed, since it’s the same judgement we’ve written about before: not whether a developer uses AI tools, but what they catch before it ships.