The AI Record
Asked to name sensitive risk domains in prose, the model claimed authorization in 38% of reviews, while about 18% of the changes touched anything authorization-shaped; required to quote the specific changed line instead, it produced 570 claims, of which 98% resolved to real code—the same model over the same pull requests, with only the form of the question changed.

Dana DzikEngineer, Coval (author of the Coval blog post on its code reviewer) · Practitioner, Coval

Post: Notes on Building a Code Reviewer, Coval blog, September 10, 2026, at The rebuild.

  • 98% claims resolving to real code, 570 claims when the reviewer must quote the changed line. A person approves

Checked against the source [ Grade B ] Said in the last year Sells AI tools

Filed under

Who is speaking

Dana Dzik, Engineer, Coval (author of the Coval blog post on its code reviewer). Builds and runs the reviewer described in the post and reports its measurements; an insider.

Coval sells AI or agent tools or services, so it has a reason to publish flattering numbers about AI. The passage stays in the archive; read it with that in mind.

How this was checked

Checked against the source by a first reader, then confirmed independently by a second reader who fetched the source without seeing the first reader's notes (Independent review agent (Claude Sonnet), blind second pass from passage text and citation only, no repository access, 2026-11-07).

Exact match in the curl-fetched text, including the em dash. The post is bylined Dana Dzik, Coval blog, September 10, 2026.

How to cite this passage

Dana Dzik (Coval), “Notes on Building a Code Reviewer”, Coval blog, 2026-09-10, at The rebuild. https://www.coval.ai/blog/how-we-built-our-code-reviewer/ Archived at https://the-ai-record.com/passages/coval-asked-name-sensitive-risk-domains-2026

More from the same source

The configuration that catches the most of what our human reviewers catch is also the cheapest one we have run, and most of its budget goes to models that would be dismissed on sight as too weak for code review—which, individually, they are.

Dana DzikEngineer, Coval (author of the Coval blog post on its code reviewer) · Practitioner, Coval

Post: Notes on Building a Code Reviewer, Coval blog, September 10, 2026, at Where it goes.

Checked against the source [ Grade B ] Said in the last year Sells AI tools Passage page

Reviews are capped at three findings, a partly arbitrary number kept because it measurably changes whether people read the output; and silence is the usual outcome, 922 of the 1,732 recorded runs of this design having published nothing, a mean of 0.70 findings per review.

Dana DzikEngineer, Coval (author of the Coval blog post on its code reviewer) · Practitioner, Coval

Post: Notes on Building a Code Reviewer, Coval blog, September 10, 2026, at The rebuild.

  • 922 review runs publishing nothing, 1,732 recorded runs of the current design. A person approves

Checked against the source [ Grade B ] Said in the last year Sells AI tools Passage page

In its first week in production the roster brought the median cost per review from about thirty cents to eleven while publishing at roughly the same rate.

Dana DzikEngineer, Coval (author of the Coval blog post on its code reviewer) · Practitioner, Coval

Post: Notes on Building a Code Reviewer, Coval blog, September 10, 2026, at The roster.

  • $0.11 median review cost, first week of the roster in production. A person approves

Checked against the source [ Grade B ] Said in the last year Sells AI tools Passage page

Something wrong? Report a correction. JSON: /api/v1/passages/coval-asked-name-sensitive-risk-domains-2026.json