Counting pull requests will hide the rot AI is putting in your codebase

Evan Merz says he was laid off from Meta after resisting AI tooling a little more than his peers, even though he generated at least 90% of his code with it in his last months there. His blog post lists four ways AI makes codebases fragile and four symptoms to look for. It is mostly anecdote, with no incident data behind it. I still think he is right, and the evidence is in other people’s studies.

A codebase’s structure lives in the heads of the people who built it. It gets there through the slow part of programming: guessing wrong, chasing a bug through several modules, rewriting the thing from scratch. An agent does that work in a context window and hands you the diff. The diff works, but nobody on the team went through the steps that would let them explain it.

Anthropic ran a randomised trial of 52 junior engineers learning a new Python library. The AI-assisted group scored about 17% lower on a comprehension quiz, roughly two letter grades, and was not meaningfully faster. The biggest gap was on debugging questions.

GitClear’s data on 211 million changed lines shows what that looks like in a repository. Moved lines, the signature of refactoring, fell from about 25% in 2021 to under 10% in 2024. Duplicated blocks of five or more lines rose roughly eightfold in 2024. Moving code requires knowing where the existing function lives and why it is shaped as it is, and pasting a copy skips that step. Merz’s example of a very good programmer reaching for JavaScript mouseover events instead of :hover fits this. It is what review looks like when each pull request arrives faster than the last.

The delivery numbers agree. The 2024 DORA report associated a 25% increase in AI adoption with a 1.5% drop in delivery throughput and a 7.2% drop in delivery stability. METR’s trial put 16 experienced open-source developers on 246 tasks in their own repositories. They took 19% longer with AI, after predicting a 24% speedup, and afterwards believed they had been about 20% faster. If your team’s confidence in its own velocity is the evidence for the rollout, METR has already shown how well that confidence tracks.

The best objection is the large field experiment. Across Microsoft, Accenture and a Fortune 100 company, 4,867 developers with Copilot access completed about 26% more pull requests. Google’s internal trial found tasks finished about 21% faster. Those samples dwarf METR’s 16 people, and I believe the numbers. Both studies also measured speed. A completed pull request is the numerator, and nobody in those papers had the denominator: what the code cost to debug, extend or hand to a new hire a year or two later. The same Cui et al. paper found the largest gains among junior developers, and juniors are the group in Anthropic’s trial that lost the most comprehension. Speed gains and understanding losses may be landing on the same people, and I find that pairing uncomfortable.

Stack Overflow’s 2025 survey gives the texture. 66% of developers named solutions that are “almost right, but not quite” as their top frustration, and 45% said debugging AI-generated code takes longer. Almost-right code passes the tests an agent wrote for it, which is how a team ends up with green dashboards and a weekend incident rota.

The fix is to measure the cost where it arrives. Track time to mitigate incidents, the share of changed lines that are moves rather than additions, and how many engineers can explain a given module without opening an agent. Merz’s claim is that incidents climb over a year or two, and your incident log can confirm or refute that within an afternoon. Take last quarter’s incidents, split them by whether the faulty module’s last change was agent-written, and compare median time to mitigate across the two groups. If your tooling cannot produce that split, you are running the rollout on the same self-assessment METR caught out, where developers who were 19% slower believed they were about 20% faster.

Prompted by AI usage generates a fragile codebase, evanxmerz.com.