AI and engineering throughput: the honest numbers

In controlled trials, AI is real but bounded. A four-thousand-developer randomized trial found Copilot users completed 26.08% more pull requests per week, and McKinsey clocked up to 2x on simple tasks. Those gains collapse on hard work, and one 2025 RCT found experienced developers 19% slower while they felt 20% faster. The honest answer is a range, not a headline: AI amplifies teams that already own their context and barely moves fragmented ones.

What do the controlled trials actually measure?

The strongest evidence for AI throughput is a multi-firm randomized trial run across Microsoft, Accenture, and an anonymous Fortune 100 manufacturer with more than 4,000 developers. Copilot users merged 26.08% more pull requests per week, with the largest lift among less experienced engineers. Isolated-task lab studies land higher — single benchmark tasks finished 40 to 55% faster — because a fresh function with no surrounding system is exactly what a code model is good at.

Read the unit carefully. “Pull requests per week” is throughput of small changes, not features shipped, not incidents avoided, not revenue. A 26% lift on the coding step is a real gain, but coding is a fraction of the lead time between an idea and a customer. That gap is the whole game in enterprise delivery, and it is where most of these numbers quietly stop measuring. It is also why elite delivery organizations out-ship the median by 208x on system properties AI does not touch.

Where do the gains shrink?

McKinsey's task breakdown is the most useful map because it refuses to give one number. The savings track complexity almost linearly:

The under-10% figure on high-complexity tasks is the one to memorize, because high-complexity work is what senior engineers at a large brewer spend their week on: gnarly returnable-deposit logic, a stock reconciliation that has to agree with SAP to the cent, a delivery-sequencing edge case. That is the majority of the value and the minority of the speed-up.

Then the counterweight. In a 2025 randomized trial of 16 experienced open-source maintainers working on repositories they knew well, allowing AI made them 19% slower — while they estimated it had made them 20% faster. The perception gap is the dangerous part: a whole org can feel accelerated while its best people quietly lose time to prompting, waiting, and reviewing plausible-but-wrong output on code they already understood.

Quality drifts the same direction. GitClear's analysis of 211 million changed lines found the frequency of duplicated code blocks rose roughly eightfold in 2024, while refactoring fell from about 25% of changed lines in 2021 to under 10%. AI is very willing to paste a fifth copy of something and very reluctant to suggest deleting the other four. More lines arrived; less structure did.

Why does one tool help one team and slow another?

Put the two extremes side by side. The maintainers METR slowed down were experts on mature code they held entirely in their heads; the model could not know their repository better than they did, so it mostly added review overhead. The trial winners were often less experienced people generating new, self-contained code where the model's median knowledge exceeded theirs. Same tool, opposite sign, and the deciding variable is how much context the human already owns.

Scale that up and it explains the enterprise pattern. A small team that owns its domain end to end — the schema, the service, the deploy — gives AI a tight, coherent context to work in, and the gains compound. A large, fragmented org where a change crosses six teams, four repos, and a release calendar gives AI more surface to generate plausible code that no single person can fully vet. The tool amplifies whatever system it lands in. Drop it onto a quarterly release train and you get more code waiting the same three months; drop it behind a stack of manual sign-offs and you get more artifacts stuck in the same queue.

How do you get the RCT numbers, not the METR ones?

The gains are real; capturing them is an engineering-and-org problem, not a licensing one. What has worked for us:

• Measure your own baseline first. The published figures are other people's teams. Record lead time, review time and change-failure rate for a month before rollout, then treat adoption as an experiment with a control group rather than a foregone conclusion. If you cannot see the delta, you cannot claim it.
• Shrink the context each engineer must hold. AI pays out most where one person owns a small, coherent slice — a service, its schema, its deploy path. Small owned-context teams on a shared event bus are the structure that turns the trial numbers into your numbers, which is exactly how we scope a first pilot.
• Aim AI at the top of the curve. Point it at documentation, tests, boilerplate and unfamiliar-codebase ramp-up, where savings run 35–50%. Keep senior humans on the high-complexity work where the tool returns under 10% and the mistakes are expensive.
• Gate for the duplication failure mode. Make cloning and churn first-class review signals, reward deletion, and stop counting lines of code as output. If AI raises volume while refactoring falls, you are buying next year's maintenance bill at a discount.
• Fix the process the tool exposes. If more code just queues behind a change-advisory board, the constraint was never coding speed. AI makes the real bottleneck louder — usually approval and integration, not typing.

The interesting question for the next year is not whether AI raises throughput — the trials settle that — but whether enterprises will restructure to deserve the gains. The tool is now cheap and everywhere; owned context, small teams and a system that can actually ship what they build are not. The brewers who win the AI dividend will be the ones who spend it on the org chart, not just the IDE.