Until there's a good way to fix that gap in the data any measure of AI impact is going to be horribly flawed.
I worked with a journalist who covered tech in the 80s & 90s and she remembered that studies then didn't show improvements in productivity with the adoption of computers, despite people feeling as though they were more productive.
Reminds me of this Jeff Bezos quote:
"The thing I have noticed is when the anecdotes and the data disagree, the anecdotes are usually right. There's something wrong with the way you are measuring it."
My guess is they used # of PRs as a measure as it’s easy to obtain, while other measures are hard, may be due to other factors, etc.
FWIW I saw a similar number for myself, around 30% more PRs in the last 6 months, compared to the 6 months before that (I picked up agentic coding around at the start of the year). And a similar increase for closed issues.
In my case this clearly doesn’t translate to as much value for the organization, or rather, it’s hard to say, as many of those PRs were things I wouldn’t even have done without AI support. This means they were low priority. However, many were of the cleanup/refactor type, so they might result in speedups later.
I can believe - easily - that there's a real uplift here, but attaching any meaning to the 24% number at all is a massive overstep.
Here's a quick hack to triple your PRs landed: Land a PR, then land ANOTHER PR undoing that one when you realize it was full of bugs, then land the PR again once you realize management doesn't care about quality, they just care about the number of PRs landed.
That's not necessarily bad. It could be a sign of effective prioritization. If you're good at working on the most important thing, and suddenly find yourself 24% more productive, what extra are you working on? The things that wouldn't have quite made the cut before.
[0] https://static.ma.nu/publications/images/2013.07.12_new_york...
[1] https://newsletter.pragmaticengineer.com/i/138252015/drawing...
So, the question is not "Does it make our programmers more productive?" but "Does it make our organization faster?".
Fun fact: all the data I've seen suggests at most a 50% uplift in those metrics. And that's at the top percentiles. Its very clear that the already high performers see the greatest uplift but anyone in that meaty middle will only see incremental gains.
Also just in general, customer satisfaction, acquisition, conversion, retention, etc.
Number of completed org-level roadmap items, org-level goals achievement rate, and so on.
I also think a good one would be seeing an increase in meeting estimation, like if project was estimated to take X days with Y devs, does the use of AI increased how often you met or beat those estimates in actual time/dev effort?
And you'd want to compare that against prior years, where no AI was used, within the same org, or try going 1 quarter without AI and another with and compare quarter to quarter.
"Number of PRs merged" seems like "number of lines of code" wearing a trenchcoat, and I thought we all agreed back in the 90s that number of lines of code was a terrible measure of software productivity...
IMO trying to measure productivity gains is a fools' errand. The only thing that matters is CSAT, escaped defects, retention, cost per contact, and other metrics that measure actual business outcomes.
(Presumably, they used a t-test that only compared people against themselves.)
Interestingly, for that study (released in 2025), participants self-rated themselves as 20% more productive, but were measured as being 19% less productive.
There was a nice talk about this by one of the author's at this year's BUILD conference: https://build.microsoft.com/en-US/sessions/BRK210
There is no comparison of the complexity, or depth of those PRs with work previously done without agents. The authors themselves acknowledge that PR count rewards small, frequent PRs and that whether the additional output justifies its cost remains unanswered. We do not know whether end-to-end feature delivery became faster, whether the additional output justified the cost, or what happened to defects, maintainability, review effort, and rework.
The conclusion seems to be little more than: Microsoft engineers are using AI tools, Microsoft is spending more on AI usage, and AI vendors are earning more subscription and token revenue.
More PRs are not necessarily more value. Without quality, delivery-time, and ROI metrics, the claimed productivity gain is largely meaningless.
I had some extra usage on claude yesterday and put several ultra workflows to look for code cleanup opportunities. It made -40 tasks and PRs just like that. This is 10% of all PRs/commits in the project history, but the before and after is arguably meaningless.
And that is based on the pricing that they sell a dollar for less than a dollar so the costs may be much much more.
