Sulat.com / Writing on AIMonday, November 16, 2026
Agent Engineering/16 November 2026

Alibaba’s Open Code Review beats Claude Code on its own benchmark but finds fewer bugs

Alibaba's Open Code Review leaves file selection and comment placement to plain code. On Alibaba's own benchmark it beats Claude Code's F1 with far fewer tokens, but finds fewer of the known bugs.

Glossy green 3D Open Code Review eye-and-shield emblem floating before a monitor showing a red and green code diff.

At a glance

  • Alibaba's website lists Open Code Review ahead of Claude Code on F1 for five models on Alibaba's own AACR-Bench, with 25.1% against 11.6% on Claude-4.6-Opus, using 385K tokens a review against 5,664K.
  • With the same model, Open Code Review reports 889 comments and 301 real bugs, while Claude Code reports 5,980 comments and 435 real bugs, so it wins on precision and loses on recall.
  • The tool leaves file selection, bundling, rule matching and line placement to plain Go code, and uses the model only to read each bundle and reason about it.
  • Delegation mode lets a coding agent such as Claude Code run the review with its own model, so ocr needs no separate API key.

Alibaba has open-sourced Open Code Review, a command-line tool called ocr that sends Git changes to a language model and returns comments pinned to specific lines. On the project’s website, Alibaba’s own benchmark puts it ahead of Claude Code on F1, the score that balances false alarms against missed bugs, for every model in the comparison, while it spends fewer tokens. It also finds fewer of the known bugs than Claude Code does.

The repository is licensed Apache-2.0 and was created on 18 May 2026. InfoQ reported the open-sourcing on 20 September, and the project’s release feed shows v1.13.9 on 13 November.

What Open Code Review is

The README says the tool began as Alibaba Group’s internal official AI code review assistant, and that over two years it served tens of thousands of developers and identified millions of code defects. The website lists 20,000-plus active users and more than 1 million review tasks.

It’s a Go program that installs through npm as @alibaba-group/open-code-review and needs Git 2.41 or later. It can review the changes in your working tree, the difference between two branches, a single commit, or whole files with ocr scan when there’s no useful diff. It speaks the Anthropic Messages API and the OpenAI Chat Completions API, and ships presets for Anthropic, OpenAI, DashScope, DeepSeek and Z.AI.

How a review runs

The README names three problems with using a general-purpose agent such as Claude Code with a review skill: it skips files on large changes, reports line numbers that drift off the real location, and gives results that swing with small prompt changes. Its diagnosis is that “a purely language-driven architecture lacks hard constraints on the review process.” Open Code Review splits the job so that plain code handles the steps that must not go wrong and the model handles the reading.

Four-step flow: code picks files, code bundles and matches rules, a model reads each bundle, separate checks fix the line.
Plain code makes the first two decisions; the model only reads.

Code does the first two steps. It decides which files need review and which to filter out, then groups related files into bundles, for example a message_en.properties file and its message_zh.properties twin. Each bundle goes to its own sub-agent, a separate model session with its own context, and the website says the tool runs 8 of them in parallel by default. Code also matches review rules to each file. According to the website, the built-in rules cover more than 10 languages, among them Java, TypeScript, Go, Python, Kotlin, Rust, C++ and C, with specific rules for null-pointer errors, thread safety, cross-site scripting and SQL injection.

The model does the reading. The README says the agent can read full files, search the codebase and inspect other changed files, and that its toolset was distilled from tool-call traces in production data at Alibaba. A sample run on the website shows calls named file_read and code_search. For long reviews the website describes a three-part memory, split into frozen, compressed and active context.

Two further modules sit outside the main agent. The website says an independent positioning module uses a three-tier progressive strategy with a model to pin each comment to exact line numbers, and a separate reflection module intercepts hallucinations and knowledge drift early.

What a comment looks like

The project’s agent skill file lists the fields in each comment: the file path, the comment text, a start and end line, a category, a severity and an optional suggested fix. Categories are bug, security, performance, maintainability, test, style, documentation and other, and severities run from critical to low. A start and end line of 0 means positioning failed. Reviews run as resumable sessions, can be written out as JSON, and can be replayed in a browser viewer where you mark comments fixed or ignored.

The benchmark behind the claim

The numbers come from AACR-Bench, a dataset from Alibaba’s Aone team that’s published on Hugging Face and described in an arXiv paper posted on 27 January 2026. The README says it covers 200 real pull requests from 50 open-source repositories in 10 languages, with 1,505 ground-truth issues annotated and cross-validated by more than 80 senior engineers. The paper says its annotation pipeline pairs AI suggestions with expert checks and raised defect coverage by 285% over the original pull request comments.

Three scores matter. Precision is the share of reported issues that are real. Recall is the share of real issues that were found. F1 is the harmonic mean of the two, which punishes a tool that’s strong on one and weak on the other.

Bar chart: Open Code Review beats Claude Code on F1 with the same model, 25.1% to 11.6% on Opus 4.6, in Alibaba's benchmark.
On Alibaba’s benchmark, the same model scores higher inside Open Code Review.

The website’s table compares Open Code Review v1.3.1 with Claude Code v2.1.169 running /code-review and Codex v0.140.0 running /review, using the same model on each side. Open Code Review’s best result is 25.1% F1 with Claude-4.6-Opus, which means 301 of the 1,505 known issues found. Claude Code’s best is 14.13% with Claude-4.8-Opus. Alibaba’s own Qwen3.7-Max placed second in Open Code Review at 21.2%.

ModelCompared withF1 (OCR / other)Recall (OCR / other)Tokens per review (OCR / other)
Claude-4.6-OpusClaude Code25.1% / 11.6%20.0% / 28.9%385K / 5,664K
Qwen3.7-MaxClaude Code21.2% / 12.2%18.3% / 23.4%625K / 5,153K
GPT-5.5Codex21.0% / 8.4%15.5% / 4.9%422K / 525K
GLM-5.1Claude Code20.4% / 11.9%15.7% / 20.8%743K / 4,038K
Claude-4.8-OpusClaude Code17.9% / 14.1%11.7% / 12.7%352K / 2,062K
DeepSeek-V4-ProClaude Code17.9% / 10.9%12.7% / 16.1%394K / 5,450K
Source: the Open Code Review website’s benchmark table. OCR is Open Code Review.

The README summarises the token result as about one ninth of Claude Code’s. Dividing the website’s token columns gives ratios from 5.4 to 14.7 against Claude Code, and 1.2 against Codex. The site’s token tile cites 1,000 pull requests, while the benchmark covers 200.

Precision over recall, by design

Against Claude Code, Open Code Review’s recall is lower in all five pairings. The README says so directly and calls it “a deliberate trade-off favoring precision over noise.” With Claude-4.6-Opus, the website’s table shows Open Code Review reporting 889 comments, 301 of them matching a known issue, and Claude Code reporting 5,980 comments, 435 of them matching.

Bars: Open Code Review made 889 comments with 301 real; Claude Code made 5,980 comments with 435 real, on the same model.
Claude Code found more real bugs and buried them in nearly seven times the comments.

That’s the trade a team signs up for. A reviewer who has to read 5,980 comments to find 435 real ones spends most of the time triaging, and one who reads 889 to find 301 spends less but misses 1,204 known issues. The project’s roadmap lists an opt-in Ultra Mode for the second half of 2026 that trades more tokens and time for higher recall, aimed at security-sensitive changes.

What outside reviewers say

Alibaba built the benchmark, ran the comparison and publishes the table. InfoQ quotes Tom Rochette, a senior developer at Shopify, writing that the one independent benchmark run so far found about 12% precision on 10 Martian-benchmark pull requests, which the maintainer disputed as a tool-call anomaly and fixed, with no independent validation since. InfoQ also quotes Daniel Vaughan, head of forward deployed engineering at HCLTech, saying the best configuration reaches 20% recall, so 80% of expert-identified issues go unfound, and that the same fixed dispatch that drives precision limits discovery of cross-file and architectural issues.

Martian’s code review leaderboard, which scores tools across 17,422 pull requests, lists 14 tools including Greptile, CodeRabbit and GitHub Copilot, and doesn’t list Open Code Review.

Where the model runs and where your code goes

In the default mode, ocr calls the model endpoint you configure. The README says that uses OCR’s own provider settings, not a Claude Code Pro or Team login, so the changed files go to whichever endpoint you pick. The project’s security assurance case says the only outbound requests are to that endpoint, that API keys are never logged, and that the browser viewer binds to localhost by default.

Two columns: default mode calls the model you configure; delegation mode hands the review to your coding agent's own model.
Delegation mode needs no separate API key for ocr.

Delegation mode flips that. The roadmap describes ocr resolving the review scope, applying excludes, loading rules and collecting diffs, then handing a structured task to the host coding agent, such as Claude Code, to run with its own model and login. The README lists plugins for Claude Code, Codex, Cursor, Kimi Code and OpenCode, and the v1.13.8 release notes add a GitHub Copilot CLI integration. For pipelines, the README lists GitHub Actions, GitLab CI, GitFlic CI and Gerrit, plus an MCP server and OpenTelemetry output.

Run it

  1. Install it with Git 2.41 or later on the machine.
  2. Pick a provider and model, then test the connection. The website says that if the Claude Code variables ANTHROPIC_BASE_URL, ANTHROPIC_AUTH_TOKEN and ANTHROPIC_MODEL are already set, ocr reads them.
  3. Run a review from inside your repository.
npm install -g @alibaba-group/open-code-review

ocr config provider
ocr config model
ocr llm test

ocr review                              # working tree changes
ocr review --from main --to my-branch   # a branch range
ocr review --commit abc123              # one commit
ocr scan --path internal/agent          # whole files, no diff needed
ocr review --format json --output result.json

The commands come from the README, which links the full documentation. An interrupted review can be resumed with ocr session list and the --resume flag.

The roadmap also lists an experimental ACP server adapter, available only from local source builds, and domain-specific long-term memory for the first half of 2027. The project shipped v1.13.8 on 12 November and v1.13.9 on 13 November.


Discover more from Sulat AI & Tech News

Subscribe to get the latest posts sent to your email.