A field study of Microsoft engineers found that early adopters of command-line AI coding agents merged roughly 24% more pull requests than a modeled counterfactual over a four-month rollout window.
That is one of the largest enterprise telemetry studies yet on agentic developer tools. It is also easy to overstate. The paper measures merged pull requests, not software quality, customer value, engineering profitability, or hours saved. Its authors explicitly warn that a merged PR is only a proxy for output.
The useful result is not “AI makes developers 24% more productive.” It is a more specific signal: active engineers who adopted Claude Code or GitHub Copilot CLI at Microsoft increased one observable form of code throughput, and the measured lift did not disappear during the study window.
What Microsoft actually studied
Microsoft researchers Emerson Murphy-Hill, Jenna Butler, and Alexandra Savelieva analyzed tens of thousands of engineers during the company’s early-2026 rollout of sanctioned command-line agents.
The study covers two related questions:
- Who tried Copilot CLI, and who continued using it?
- What happened to merged-PR output among engineers using Copilot CLI or Claude Code?
The adoption analysis focused on Copilot CLI because it had a clearer population of eligible users. The outcomes analysis included both tools. Microsoft engineers already had access to IDE-based Copilot features, so the study measures the incremental adoption of command-line agents rather than first exposure to AI assistance.
The post-rollout observation window ran from January 5 through April 29, 2026. For its main outcome, the paper counted a pull request as merged when it completed within 28 days of creation and limited the cohort to engineers with at least two merged PRs during a four-week pre-rollout period.
Where the 24% number comes from
The researchers used a Bayesian structural time-series method to estimate how many PRs the early-adopter cohort would have merged without adopting the tools. The synthetic counterfactual was built from engineers who created PRs but never used either command-line agent.
The model estimated a 24.0% increase in merged PRs per engineer per day, with a 95% credible interval from 14.5% to 33.7%.
Figure: The solid line shows observed daily merged PRs among adopters; the dashed line is the Bayesian structural time-series counterfactual. Source: Murphy-Hill, Butler, and Savelieva, CC BY 4.0.
The estimated lift persisted across the available window. The paper reports a 29.4% point estimate for February and 20.0% across March and April. Their credible intervals overlap, so the apparent decline is not statistically distinguishable from ordinary variation in this analysis.
That persistence matters because some earlier studies of AI developer tools found an initial increase that faded after a few months. Four months is still not a long-term result, but it is enough to make a pure first-week novelty explanation less convincing.
Heavier use tracked with more PRs
The paper also compared the same engineer’s high-use and zero-use weeks. This within-person design controls for stable individual characteristics such as role, skill, team, and coding style.
Merged-PR output rose monotonically with days of agent use:
- Three tool-use days in a week were associated with 15.0% more merged PRs than that engineer’s zero-use weeks.
- Five or more tool-use days were associated with 50.1% more merged PRs.
This is a dose-response association, not definitive proof of causation. Engineers may use agents more often during weeks filled with smaller or more automatable tasks. They may also open an agent because they already expect to ship more work that week.
Still, the pattern is harder to dismiss than a satisfaction survey. It is based on direct tool telemetry and merged work rather than asking developers whether they felt faster.
The surprising Copilot CLI comparison
Among engineers who used only one of the two tools, the study found different within-person lifts:
- Copilot CLI use was associated with 24.9% more merged PRs in use weeks.
- Claude Code use was associated with 11.4% more merged PRs in use weeks.
The paper describes the difference as surprising because public developer sentiment often rated Claude Code highly for autonomous work. The result should not be read as a clean benchmark showing that Copilot CLI is the better coding agent.
Tool access was not randomly assigned. The products had different rollout paths, user populations, organizational support, task mixes, and usage policies. Microsoft also owns GitHub, the developer of Copilot CLI, and announced near the end of the observation window that most internal Claude Code licenses would later be discontinued. The researchers ended the window before that migration to avoid contaminating the comparison, but selection effects remain.
The tool-specific numbers are best treated as an internal field observation, not a general model ranking.
Adoption spread through coworkers
The strongest predictors of trying Copilot CLI were social rather than demographic.
An engineer had 54% higher odds of first use when at least one quarter of their regular code-review peers had recently used the tool. Exposure among coworkers sharing the same skip-level manager was associated with 216% higher odds of first use at the highest exposure level. Having a direct manager who used Copilot CLI was associated with 82% higher odds.
These are changes in odds, not percentage-point changes in adoption probability. The design also cannot fully separate peer influence from homophily: similar engineers may cluster together and adopt similar tools without directly persuading one another.
The rollout lesson is still useful. A license announcement alone is unlikely to build sustained adoption. Engineers learn agent workflows by seeing task decomposition, review habits, reusable prompts, and successful changes inside their team.
Interestingly, heavy prior IDE Copilot users were more likely to try the CLI but less likely to retain it. The authors suggest those engineers may have had a familiar IDE alternative to return to, while first-time AI-tool users who found value in the CLI had no competing habit.
Why merged PRs are not productivity
PR count rewards a particular shape of work. It can increase because engineers deliver more useful changes, but also because they split work into smaller PRs, automate low-value cleanup, or create additional review load.
The study does not directly measure:
- Defect rates or production incidents.
- Reverted changes and long-term maintenance cost.
- Code complexity, duplication, or security regressions.
- Reviewer time and cognitive load.
- Customer outcomes or business value.
- Total inference cost per accepted change.
The authors acknowledge several additional limitations. Adoption was not randomized, heavier-use weeks may have easier task mixes, the data comes from one company, Azure DevOps PRs do not capture all engineering work, and the authors are Microsoft employees studying a product made by Microsoft-owned GitHub.
None of those limitations invalidate the observed signal. They define its boundary.
How an engineering organization should test the result
A company considering a large coding-agent rollout should reproduce the measurement on its own workflow rather than importing the 24% figure into an ROI model.
Track at least four layers:
- Throughput: merged changes, cycle time, task completion, and work-in-progress.
- Quality: defects, rollbacks, review rounds, security findings, and maintenance changes.
- Economics: model and tool spend, retries, CI usage, and reviewer time per accepted change.
- Adoption: first use, active days, retention, task classes, and team-level diffusion.
Use versioned task and outcome definitions. Compare teams or engineers over time, but document changes in release schedules, staffing, codebase ownership, and project mix that could move the metrics independently of AI use.
The Microsoft data also suggests a rollout tactic: create visible examples inside real teams. A small group of respected engineers demonstrating how they constrain, test, and review agents may produce more durable adoption than mandatory training or a company-wide license email.
The signal
Command-line coding agents appear capable of changing real engineering throughput at enterprise scale. The Microsoft study provides stronger evidence than benchmark scores or self-reported time savings because it links direct usage telemetry to merged work over several months.
Its headline number remains narrower than the marketing version. The study found more merged PRs among adopters under a specific rollout, not a universal 24% increase in engineering productivity.
The next important studies will connect agent use to defects, review burden, customer outcomes, and cost per durable change. Until then, the right conclusion is optimistic but measured: coding agents can move output, and organizations still need to prove that the additional output is worth owning.
