Precision Beats Recall: Why the Cheaper Model Wrote the Better Code Review

We expected this one to be boring. GPT-6.1 Terra is the newer, larger sibling of GPT-6 Luna, and it costs about 8 times as much per review. Put both through the same code review benchmark and the bigger model should win. It didn't. Terra scored an F1 of 33.5. Luna scored 49.4.

What made us dig in was how it lost. Terra didn't miss the bugs Luna caught. It caught them too, and then kept talking.

How we scored it

The pull requests come from Code Review Bench, the open dataset behind our benchmark write-up. Each one ships with an answer key: the bugs human reviewers confirmed were really there. Both models went through the same review pipeline with the same prompt, the same repository context and the same reasoning effort. We ran it several times, so no single lucky pass could carry the result.

An LLM judge then lines up every issue a model raised against the answer key. Recall is the share of known bugs a model found. Precision is the share of what it said that turned out to be a known bug. F1 balances the two.

Same bugs, half the precision

Recall landed within two points: 45.5% for Luna, 43.2% for Terra, and mostly the same bugs. Precision is where they split. Just over half of what Luna raised matched a known bug. For Terra, it was just over a quarter.

The cause is volume. Luna raised 3.7 issues per review on average, Terra about 7. Nearly all of the difference went somewhere the answer key doesn't cover.

Is Terra just wrong, then?

Not exactly, and this is the interesting part. Answer keys are incomplete. A reviewer who wrote down four bugs in a pull request never promised there were only four. So we read every issue both models raised and sorted each one by hand: a known bug, something both models flagged on their own, a defect only one model claimed, or noise.

When Luna went beyond the answer key, Terra had flagged the same thing 76% of the time. Two models of different sizes landing on the same unlisted issue is a good sign the issue is real and the key missed it. When Terra went beyond the key, Luna agreed only 28% of the time.

The rest of Terra's extras split in two. The first group is concrete defects nobody else raised. A few were real. On one pull request it caught an authentication edge case the original reviewers had missed, which is the honest case for a more thorough model. Most were plausible, hard to verify, and far from what the change was trying to do.

The second group is noise. On one pull request Terra asked for a cap and a rate limit on a guest-invite endpoint, in case someone used it to send bulk email. On the same change it warned that two calendar updates could land out of order and overwrite each other. Elsewhere it flagged a password toggle that keyboard users can't tab to, and posted the same unchecked database lookup twice in one migration.

A careful senior engineer could have written any of these. Posted together on every pull request, they stop reading like judgement and start reading like a checklist.

The agent reads the review first now

A couple of years ago, a low-value comment cost a human about ten seconds to read and resolve. Recall mattered more than precision, because a missed bug shipped and a noisy comment didn't.

Plenty of teams don't work that way anymore. Claude Code, Codex and tools like them can watch a pull request, read each review comment and push a fix for it, round after round, until the reviewer goes quiet. The agent doesn't triage. Every comment is a task.

Run those comments through that loop. The guests endpoint gets a rate limiter and a setting to configure it. The calendar code gets a lock. The toggle gets its tab stop back, and the migration grows a second defensive branch. None of it was the point of the pull request. All of it is new code, and new code is what the reviewer reads in the next round. A focused change turns into a sprawling one, and whoever finally reviews it has to work out which parts were ever needed.

So which matters more?

It depends on what happens to a finding after it's reported. In pull request review, precision wins. Comments land in the middle of someone's work, more and more of them are acted on automatically, and each one spends the attention of the person who has to merge. Three issues that matter beat those same three plus four that don't, even if one of the four turns out to be real.

A scheduled audit of a whole repository is the opposite. Findings go into a backlog, someone triages them on their own time, and nothing is auto-fixed into a branch that has to ship today. That's why we rank our security scanner on a metric that weights recall. A thorough model earns its price there.

If you're choosing an AI reviewer, ask for its precision as well as its recall, and look at how many comments it leaves on an ordinary pull request. Then read a few and ask whether you'd let an agent act on them unsupervised. In our runs, the model that cost an eighth as much was the better reviewer. It knew when to stop.