Benchmarking Patchlight: Our Scores on Code Review Bench

We ran Patchlight through Martian's open-source Code Review Bench — 50 real PRs, 137 human-verified findings, one judge for every tool. Here are the numbers, the methodology, and where we're strong.

July 14, 20269 min readPatchlight Team · Research

We believe a code review tool should publish its scores on an independent benchmark — not a hand-picked demo reel. So we ran Patchlight through Code Review Bench, the open-source benchmark for AI code review built by Martian, and we're publishing exactly what came out: an F1 of 52.1, placing Patchlight third among the five frontier tools we evaluated on the offline dataset — ahead of Augment and Qodo v2, behind Cubic v2 and Qodo Extended.

52.1

F1 score

harmonic mean of precision & recall

59.1%

Recall

81 of 137 golden findings caught

46.6%

Precision

share of our comments matching a golden finding

×2

Reproduced

identical 52.1 across two independent full runs

Every number in this post comes from the benchmark's public pipeline, scored with one consistent judge configuration across every tool, and reproduced end-to-end twice. Here's the full picture, how the evaluation works, and where our reviewer is strongest today.

The leaderboard

Code Review Bench's offline dataset is 50 real pull requests from five major open-source projects — Sentry, Grafana, Cal.com, Discourse, and Keycloak — each annotated with human-verified “golden” findings: the 137 real issues a reviewer should have caught. A tool is scored on how many of those it finds (recall), and how much of what it says corresponds to a real golden finding (precision). F1 combines the two. Toggle the metric below; hover any bar for the full breakdown.

Code Review Bench — offline leaderboard50 PRs, 137 golden findings, single judge configuration across all tools. Amber = Patchlight.

A note on fairness: these are our own measurements, produced by running the benchmark's open pipeline on each tool's reviews of the same 50 PRs, under the same judge and the same matching rules. Nothing here is Patchlight-flavored — the dataset, the golden findings, and the judge prompts are all published by Martian, and you can reproduce every number from the repository. The official, continuously updated leaderboard lives at codereview.withmartian.com.

How the evaluation actually works

Most code review “benchmarks” are a vendor grading its own homework. Code Review Bench is designed differently, and it's worth spelling out the mechanics, because they're what make the scores meaningful:

Real PRs, real bugs. The 50 pull requests are drawn from production codebases in five languages — Python, Go, TypeScript, Ruby, and Java — ten PRs per language. The golden findings were curated and human-verified, each with a severity label from Low to Critical.
An LLM judge does the matching. Each review comment a tool produces is compared against the golden findings by a judge model working from published prompts. A comment that matches a golden finding is a true positive; a golden finding no comment matches is a miss; a comment matching nothing is counted against precision.
One judge for everyone. We scored all five tools with a single judge configuration, validated at 93% agreement with the benchmark's reference judge on this dataset. Judge choice shifts absolute numbers, which is exactly why we never compare scores across different judges — every number in this post comes from the same one.
Reproducibility over cherry-picking. LLM systems are stochastic, so a single lucky run can flatter you by several points. We ran the full pipeline — fresh reviews on all 50 PRs, fresh scoring — twice, independently. Both runs produced an F1 of 52.1. We report nothing we couldn't reproduce.

Precision, recall, and the trade every tool makes

A single score hides the most interesting decision in code review: when the tool isn't sure, should it speak up or stay quiet? Plotting precision against recall makes each tool's posture visible.

Precision vs. recall — every tool makes a different tradeDashed curves are constant-F1 contours. Up and to the right is better on both axes.

Patchlight sits deliberately on the higher-recall side of its F1 contour: we catch 59% of the golden findings, and we'd rather surface a real Critical bug alongside a debatable nit than miss it. We think that's the right default for a reviewer whose findings arrive ranked by severity — you can skim past a weak Low-severity comment in two seconds, but a missed authentication bug ships. That said, precision is where the gap to the leaders lives, and it's our top improvement target.

Where Patchlight is strong

Consistency across stacks. Patchlight has the tightest cross-language spread of any tool on the board: 8 F1 points between our best language (Java, 56.0) and our worst (Go, 48.0). The field's top tool swings by more than 21 points depending on the language. If your organization runs a polyglot codebase, the score you see on the leaderboard is close to the score you'll get on your repo — whichever repo it is. On Ruby, historically the hardest language for AI reviewers, we're within a single point of the best tool measured.

F1 by language — Patchlight vs. the fieldGray band spans the lowest to highest score among all five tools on that language (10 PRs each). Amber dot = Patchlight.

The subtle stuff. The benchmark labels each PR by how hard its bugs are to spot and how risky the change is. Patchlight's profile tilts toward exactly the cases you want a machine reviewer for: our score rises as bugs get more subtle and as the blast radius of the PR grows.

Patchlight F1 by bug subtlety and PR riskBenchmark labels per PR. Small slices (1–5 PRs) are directional, not statistically firm — sample sizes shown under each bar.

The small slices come with an honesty caveat — “very subtle” is only 3 PRs and “critical risk” only 5, so treat those bars as directional. But the shape is consistent with what we optimize for: on the 36 PRs labeled “subtle”, the biggest slice of the dataset, we score above our overall average, and on critical-risk PRs we're the second-strongest tool measured.

Security findings, release over release. The latest Patchlight engine materially improved on security-class golden findings — origin-validation bypasses, overly permissive framing policies, unsafe comparison patterns — moving from catching 1 of the 9 security goldens to 5 of 9. Security issues are rare in the dataset, so this barely moves the headline F1, but it's the class where a single miss costs the most.

A steep improvement curve. Under the identical dataset, judge, and scoring recipe, the previous Patchlight release scored 48.0. The engine you get today scores 52.1 — a gain of 4.1 F1 in one release cycle, driven by a research loop we run against this benchmark: we autopsy every missed finding, classify why it was missed, and ship targeted fixes rather than vibes. (We're keeping the specific techniques to ourselves, but the process is simple to state: measure, diagnose, fix, re-measure — and never trust a gain you can't reproduce.)

What we're working on next

The leaderboard is honest about our gaps, so we will be too. Precision (46.6%) trails the two tools ahead of us, and small PRs are our weakest size class — counterintuitively, we do better on medium and large changes than on ten-line diffs. Both are active workstreams, and because our benchmark runs are cheap and reproducible, you should expect updated numbers here as releases land rather than an annual marketing refresh.

Run it yourself

Everything needed to check our math is public: the PRs, the golden findings, the judge prompts, and the scoring pipeline are in the code-review-benchmark repository, and the official leaderboard with additional tools is at codereview.withmartian.com. If you build a code review tool, run it — the field gets better when nobody grades their own homework. And if you want to see what a 52.1 feels like on your own pull requests, Patchlight reviews your first PRs for free.