Does AI code review catch real problems?

“Before the changes, authors acted on 34% of the code quality reviewer's serious findings. Now it's 59%.”

Ryan SkidmoreAuthor of Cloudflare's "Orchestrating AI Code Review at scale" blog post · Practitioner, Cloudflare

“We measure noise by watching what people do with the comments. It's easy to catch more bugs if you don't mind flagging a lot of non-bugs, so we need a check on that. We sort every comment the reviewer leaves by how the person on the PR responded: they fixed it, said it was wrong, said the behavior was intentional, or decided it didn't matter. We also run the reviewer on PRs we know are clean and count how often it reports something that would have led someone to make a harmful change.”

AnthropicPublished by the organization itself (post), no named author

“Findings get verified by other agents before humans review them. Several agents look for bugs at the same time. Then a separate group votes on each finding, and it only gets posted if enough of them agree it's real. Findings that are only about style, diagnostics, or cleanup get dropped. We A/B tested those filters on our own PRs before keeping them.”

AnthropicPublished by the organization itself (post), no named author

“Reviews are capped at three findings, a partly arbitrary number kept because it measurably changes whether people read the output; and silence is the usual outcome, 922 of the 1,732 recorded runs of this design having published nothing, a mean of 0.70 findings per review.”

Dana DzikEngineer, Coval (author of the Coval blog post on its code reviewer) · Practitioner, Coval

“[uReview] does … around 25,000 comments a week, and we get 10% of them actually get some feedback … We also saw that the overall address rate was around 67%, and almost three quarters of the high severity issues were usually addressed by the developers”

Ameya KetkarSpeaker, AI Engineer World's Fair 2026 (job title not stated) · Practitioner, Uber

“A lot of our code is now written with an AI assistant in the loop. The first thing we tried was the obvious thing: ask the same Claude session that wrote the change to review it. That does not work. It missed critical bugs, and it missed them confidently, because it had already reasoned its way to believing the code was correct, and re-reading its own work only confirmed the conclusion.”

Arthur VinbergHead of DevOps & IT · Leader, Ionix

“We already had Ona's code review automation running on every PR. The agent reviewed diffs, ran the code in a full environment, cross-referenced tickets, and left inline comments with an 88% acceptance rate and a 50% reduction in time spent on pull requests.”

OnaPublished by the organization itself (post), no named author

“We've been running Code Review internally for months: on large PRs (over 1,000 lines changed), 84% get findings, averaging 7.5 issues. On small PRs under 50 lines, that drops to 31%, averaging 0.5 issues. Engineers largely agree with what it surfaces: less than 1% of findings are marked incorrect.”

AnthropicPublished by the organization itself (post), no named author

“we conclude that RovoDev Code Reviewer is effective in generating code review comments that could lead to code resolution for 38.70% (i.e., comments that triggered code changes in the subsequent commits)”

AtlassianPublished by the organization itself (paper), no named author

“developers are most likely to resolve comments about readability, bugs, and maintainability. These comment types are found to be more specific and actionable, i.e., code changes were made to resolve the comments, with resolution rates between 36% and 43%.”

AtlassianPublished by the organization itself (post), no named author

“uReview today analyzes over 90% of the weekly ~65,000 diffs (equivalent of pull requests) landed at Uber. Engineers who interact with the tool mark 75% of its comments as useful”

Sonal MahajanStaff Engineer, Programming Systems team · Practitioner, Uber

“Important findings per review went from 0.28 before to 0.30 now, and the intervals overlap.”

Ryan SkidmoreAuthor of Cloudflare's "Orchestrating AI Code Review at scale" blog post · Practitioner, Cloudflare

“Medium effort did lose real findings. In the randomized split, code quality at medium led to roughly a quarter as many acted-on findings as at high, which is why we're moving it to high.”

Ryan SkidmoreAuthor of Cloudflare's "Orchestrating AI Code Review at scale" blog post · Practitioner, Cloudflare

“Review has to converge. An agent author and an agent reviewer can keep each other busy indefinitely. One CLI PR went through more than fifty rounds of review and fixes overnight before its author stopped it. When we ran git blame on what the reviewer kept flagging, about 59% of it was code that had been there since the reviewer's first pass.”

AnthropicPublished by the organization itself (post), no named author

“Any incidents are now an eval. The team collected past PRs that caused incidents, with their fix-forwards and reverts, and had engineers who know that code confirm each bug and where it lived. The reviewer runs against that set daily.”

AnthropicPublished by the organization itself (post), no named author

“Before and after the convergence work: pull requests passing after one review and fix round rose from 65% to 78%; pull requests needing five or more rounds fell from 8.9% to 1.5%; mean reviews to pass fell from 2.08 to 1.32.”

AnthropicPublished by the organization itself (post), no named author

“Noise costs more when the author is an agent. A person can wave off a nit in seconds. An agent fixes it, which means another commit and another round of review.”

AnthropicPublished by the organization itself (post), no named author

“When we launched the research preview, the share of internal PRs getting substantive review comments went from 16% to 54%. Most PRs over 1,000 lines get findings, and most small ones come back clean.”

AnthropicPublished by the organization itself (post), no named author

“A review bot that posts on every pull request can waste more attention than it saves. Humans have to read the comments, decide which are right, and resolve threads. A reviewer who has learned that the bot is often wrong learns to skim, and then skims the good findings too.”

TradersPostPublished by the organization itself (post), no named author

“Asked to name sensitive risk domains in prose, the model claimed authorization in 38% of reviews, while about 18% of the changes touched anything authorization-shaped; required to quote the specific changed line instead, it produced 570 claims, of which 98% resolved to real code—the same model over the same pull requests, with only the form of the question changed.”

Dana DzikEngineer, Coval (author of the Coval blog post on its code reviewer) · Practitioner, Coval

“The configuration that catches the most of what our human reviewers catch is also the cheapest one we have run, and most of its budget goes to models that would be dismissed on sight as too weak for code review—which, individually, they are.”

Dana DzikEngineer, Coval (author of the Coval blog post on its code reviewer) · Practitioner, Coval

“One of the biggest learnings in this process was like the model doesn't know that it's wrong. It always confidently says 100% sure that, yeah, this is the review for your code. Go ahead. But we saw that, no, it actually needs a lot of guidance from the teams, because each team has its own style guide, its own patterns or … anti-patterns that they want to look for.”

Ameya KetkarSpeaker, AI Engineer World's Fair 2026 (job title not stated) · Practitioner, Uber

“Most days the entire output is one sentence saying the change looks fine, which is what we built it to do. On the days it leaves an inline comment, someone reads it.”

Arthur VinbergHead of DevOps & IT · Leader, Ionix

“The second is permission to find nothing. The prompt says outright that returning zero findings is the correct and expected outcome for a clean PR, and that findings must never be manufactured to look thorough.”

Arthur VinbergHead of DevOps & IT · Leader, Ionix

“So the concrete example here is we do security scanning after on our PRs in local and after our PRs even land because they will always find some things that are real that we have overlooked. And they sort of beat humans on the … code review. And it's expensive. It … us, I think like five bucks a PR or something like that to run all the checks that we want. But that's where we made an explicit decision that it's worth it.”

Greg PstruchaDeveloper, Sentry · Practitioner, Sentry

“engineers don't trust AI opinions. They just trust clear findings tied to standards that we … agreed upon. … it stopped sounding more like, maybe this is a problem and started sounding more like this violates a standard that we actually care about.”

Rajesh BhatiaSenior Director of Engineering, Cloudflare · Leader, Cloudflare

“Memory matters: Per-PR conversation context dramatically improved review quality over stateless approaches”

JuspayPublished by the organization itself (post), no named author

“In addition, Rovo Dev is focusing only on actionable comments with only 0.6% classified as “No Issue”.”

AtlassianPublished by the organization itself (post), no named author

“Atlassian’s internal projects, Rovo Dev generated a higher proportion of bug (2.8x) and maintainability (1.4x) comments, while human reviewers focused more on readability and slightly more design.”

AtlassianPublished by the organization itself (post), no named author

“so when Devin fixes his own comments, you might be scared that, oh, maybe I’ll infinite loop. But we’ve put a lot of work into making sure it doesn’t, both by making sure that the comments are high signal, but also that the agent is thoughtful about what comments it immediately goes and tries to fix, and what comments it’s like, “Wait a second, I think you’re wrong.””

Walden YanCo-founder and Chief Product Officer · Executive, Cognition

“We absolutely had to get the bots to help us review PR's. … But now, with the repo readiness work that the champions had done, coupled with better models and tools, … we were getting much better results here. So we enabled … on all the repos, and we also created an auto fix loop where if … issues, another agent will automatically fix those issues and commit them to the PR. So now, at least people were no longer complaining about not wanting to review sloppy PR's from bots, right? By time they got to them, they were in pretty good shape.”

Angie JonesLed AI enablement and the agent engineering org at Block, as she describes in the talk; now VP of Developer Experience, Agentic AI Foundation · Leader, Block

“In our case, Bugbot ran for a month and caught around ten real bugs. One of the best examples was a subtle mix-up between two different phone number lists in the code.”

Peter EysermansAuthor, madewithlove blog (role not stated on the page) · Practitioner, madewithlove

“It occasionally throws false positives. For instance, in the review above, it flagged swapped parameters that weren’t actually swapped. It’s also overly cautious about nullable types, warning about missing null checks even in safe contexts.”

Peter EysermansAuthor, madewithlove blog (role not stated on the page) · Practitioner, madewithlove

“Before switching all reviews over from v1 to v2, we ran every review in every canary repo on both v1 and v2 in a comparative shadow mode. The two agreed on the approval decision in 98% of the merge requests we tested, so we moved everyone over in one go in early October and removed the v1 code.”

Ryan SkidmoreAuthor of Cloudflare's "Orchestrating AI Code Review at scale" blog post · Practitioner, Cloudflare

“And then the self review, which is kind of interesting because we have this, we created this review pool. So we have one model, which is a Sonnet that does one review. … then we have kind of a … cheaper model that also does a review. And then we have the third model, … right now, it kind of consolidates this and verifies the findings in the reviews and decides if it should change something or not. And when this review pool is kind of finished and happy, … will assign it to a human.”

Hallstein BrøtanSenior consultant, Novanet; developer on Nordic Corporate Bank's system · Practitioner, Nordic Corporate Bank

“the best example of a rule that I can think of internally is for Bugbot. So when we're doing database migrations, we're not really using foreign keys on a database for performance reasons. And the models like the right way to do this is use foreign keys, right? So they will always add a foreign key. But when it … hits … and there's a PR created, we have Bugbot … it's like, oh, I have this rule saying like we should never use foreign keys. So then it flags this.”

Eric ZakariassonEngineer, Developer Experience and Product, Cursor · Practitioner, Cursor (Anysphere)

“Off-the-shelf AI code review tools: Great for generic checks, but couldn't understand our Haskell payment systems or custom conventions”

JuspayPublished by the organization itself (post), no named author

“Both utilize a semantic search pipeline based on retrieval-augmented generation to assemble relevant contextual information for the review, thereby tackling the uncovered challenges. Developers evaluated both variations in real-world settings: AI-led reviews are overall more preferred, while still being conditional on the reviewers’ familiarity with the code base, as well as on the severity of the pull request.”

Fannar Steinn AðalsteinssonAuthor, affiliated with WirelessCar Sweden AB and Chalmers University of Technology · Practitioner, WirelessCar