GitHub launches ReviewBench: 219 real PRs across 19 languages as an open benchmark for AI code review

On October 5, GitHub published ReviewBench on its blog, an open benchmark for AI code review agents. To avoid the self-curated dataset problem, the team analyzed 103.9M real GitHub pull requests across languages, repo size and change distribution, then filtered 219 PRs from 187 public open-source repositories covering 19 languages as the corpus. They deliberately down-weighted single-file small changes so multi-file substantive changes had higher representation.

The ground truth set was produced and de-duplicated by human reviewers, author follow-up commits, deterministic static analysis tools and multiple frontier models. Each finding is labeled with severity and category, then adjudicated by Claude Sonnet 5 under public scoring criteria for being real, relevant and non-trivial. Senior engineers independently re-annotated all ground-truth findings and achieved 96.6% agreement with ReviewBench judgments. Metrics are split into grounded (precision/recall/F1 against the gold set) and augmented (independent evaluation of unmatched findings, rewarding genuinely new findings).

GitHub used ReviewBench to evaluate Copilot code review: a multi-model integrated review round showed offline +4.45% precision and +12.88% recall, while the production A/B test showed +8.00% precision, +13.58% recall, +61% comments and +262% critical-comment share. Offline metric directions aligned with production experiments, giving product changes a reliable early signal.

Dataset, scoring prompts and self-service scorer are all public. Developers can sign in to the ReviewBench site with GitHub, register a container image and configuration, tune on a 25-PR test set, then run three rounds on the full 219-PR set scored by the unified scorer.