Kitta: Code Review Agent for Higher-Quality Code Reviews

This article introduces Kitta, a domain-customized Code Review Agent that leverages project-specific knowledge and tailored workflows to deliver expert-level code reviews.

By Hangbao

This article is excerpted from the Alibaba Cloud AI Agent Handbook. Project URL: https://github.com/aliyun/ai-agent-handbook

Now that AI-assisted programming has become widespread, code output has grown exponentially while review capacity has not kept pace. For foundational software such as kernels, PR size and review time have both surged; general-purpose AI review tools, lacking domain knowledge and an understanding of the business, cannot meet expert-level review requirements. The bottleneck in R&D efficiency is shifting rapidly from "writing" to "reviewing."

This article introduces Kitta, a Code Review Agent deeply customized for a specific domain. It turns a general-purpose agent into a domain specialist: by accumulating project-level expert assets, customizing the review workflow and the domain knowledge base, and building a closed loop of coordinated online and offline evolution, it precisely addresses domain-specific code issues.

Kitta has been running in the OpenAnolis community ever since it went online, and has completed reviews of more than 70,000 patches in total, with a gatekeeping accuracy of 97% and a merge rate stable at 62%. Its domain-customization methodology has been validated across six domains — operating-system kernels, compilers, and Agent platform applications among them — improving review quality by 17.0%–31.6% and freeing Maintainers' effort from line-by-line review so that it can go to ruling on the disputes that really matter.

Background and Current State

▍ 1.1 After the Surge in Code Volume, Quality Pressure Rises Sharply

Let us start with a few sets of numbers.

The number of CVEs fixed upstream in the Linux kernel is climbing fast: from kernel 6.9 to 6.19 the figures stayed roughly in the range of four to five hundred, then jumped to about 1,250 at 7.0 and approached 1,900 at 7.2; by 7.3 the number is projected to exceed 2,000. Widened out to the whole industry, year-over-year CVE growth is even more striking — +394% overall and +811% for high-severity vulnerabilities (source: Linux Foundation, The CRA Readiness Reality: What Changed and What Didn't Between 2025 and 2026). This growth exposes the code-quality pressure we are under: both the volume and the severity of exposed code defects keep increasing. Along with it, the amount of code that open-source projects need to merge has surged as well. In the ANCK kernel repository of the OpenAnolis community, for example, PRs merged into the branches of the two mainstream versions have continued to climb since AI Agents were introduced.

The data on R&D efficiency points to the same trend. DORA (the R&D-efficiency research program under Google Cloud) reports in its annual study State of AI-assisted Software Development that while AI takes part in development more deeply, the code bug rate rises by 9%, code review time by 91%, and average PR size by 154% (this edition surveyed about 5,000 respondents, and the figures are a correlation measure rather than a causal conclusion).

In short, in an era of human-AI co-authored coding, writing code is no longer limited by human capacity, but review capacity has not grown along with it. The productivity bottleneck is moving from "writing" to "reviewing."

▍ 1.2 The Significance of Code Review in Agentic CI

Facing this trend, the OpenAnolis community is exploring what an AI Native R&D organization looks like, and is upgrading continuous integration into Agentic Continuous Integration. The pipeline has four stages:

AI-assisted programming — R&D output rises sharply across the board; humans and AI co-author code, so writing is no longer limited by human capacity;
Intelligent code review — engineering standards and process, architectural and design consistency, change completeness and prerequisites, defects and security risks all have to be checked; but human review resources are insufficient, so AI review is imperative;
Intelligent testing — including deep vulnerability discovery, patch-directed testing, security patch verification, regression and performance testing, and further verification testing;
Community review and merge — people make the final decision.

It is clear that AI review sits in a position that connects what comes before with what follows: upstream, AI-assisted programming pushes the volume of code output up, and downstream, intelligent testing and community merging both rest on the premise that the code entering those stages is sound. If the review stage gives way, defects either leak downstream or Maintainers are overwhelmed by the flood of PRs produced by human-AI co-authoring. This is the problem Kitta is meant to solve.

The Deeper Challenges of AI Code Review

AI Code Review brings some obvious benefits, such as greatly faster response times and automated interception of basic defects. In core business, however, general-purpose AI review tools usually fall short of expert-level review requirements.

Taking this apart, today's general-purpose Code Review solutions show clear limitations on two dimensions. The first is insufficient constraint on engineering behavior: incomplete coverage, drifting locations, unstable results — on larger changes the tool tends to review only some of the files and misses the rest, the problems it reports do not line up with the actual code positions, and review quality swings widely with small differences in the prompt. The second is insufficient depth of inspection: it lacks semantic understanding of the project being reviewed, and its rules stay on the surface — it treats every project and every review the same way, and can only scan defects in the code itself, without sensing the review needs that are specific to the project. This shows up as three deeper challenges:

Domain knowledge is lacking. Technology stacks differ greatly across domains, and the body of knowledge has to be updated dynamically. General-purpose models have seen a great deal of code, but they may not understand the locking semantics of the kernel, nor the conventions of a compiler's intermediate representation — and these are exactly what decide whether a review conclusion is right or wrong.
Business understanding is inadequate. Projects develop in many different ways, so the positioning and the capabilities of a review have to fit the business scenario. For a community open-source project, an internal enterprise repository, and distribution maintenance, the standard for "what counts as a problem, and how serious it is" is completely different.
The boundary of trust is blurred. There are no objective quality assessment criteria, and no evolution mechanism that is easy to align with user needs. A user saying something is "good" does not necessarily make it good, and an Agent feeling good about itself is even less reliable — without an objective evaluation and evolution loop, trust cannot be established.

To sum it up in one sentence: when AI is deeply involved in core business, it can "understand" the code without necessarily "understanding" the project.

Technical Practice of the Kitta Code Review Agent

Against these three challenges, Kitta's approach is to domain-ize a general-purpose Agent, and to build a genuinely knowledgeable, expert-level Code Review Agent in three steps.

▍ 3.1 Step One: Managing Project-Level Expert Assets

Domain knowledge does not come from nowhere; it is distilled from real history. Kitta starts from the real reviews and defect records in the community's development history, covering multi-source data such as a project's GitHub repositories. Raw records go through collection and association, normalization and denoising, semantic parsing, and isolated reproduction, and are then sedimented into project-level expert assets; "isolated reproduction" is there to make sure that every defect pattern and every expected conclusion is verifiable and can withstand being questioned, rather than mistaking noise for experience. This asset set contains two kinds of core content:

Judgment criteria — review principles, defect patterns, and project conventions, which answer "against what standard do we judge?";
Validation samples — problem samples, normal samples, and expected conclusions, which answer "how do we measure whether the judgment was right?"

These assets are verifiable, sedimentable, and reusable: for a new project, the same pipeline can distill its real history into a new set of expert assets. The essence of this step is to turn "knowing the domain" from personal experience into an organizational capability.

▍ 3.2 Step Two: Dual-Engine Customization of Code Review

With expert assets in hand, the next thing to settle is "how to look." Kitta drives this with two wheels: the workflow definition decides where to look, and the knowledge decides how to look. Taking the ANCK deployment in the OpenAnolis community as an example, here is what each wheel does.

Customizing the review workflow, so that AI sees everything: this defines where the review should look, across dimensions such as semantic consistency, backport completeness, distribution conventions, and integration risk — semantic consistency keeps patch behavior from drifting away from what was intended; backport completeness watches whether cross-version patch sequences are complete and whether anything is missing; distribution conventions check compliance with the distribution's merge requirements; and integration risk assesses the knock-on effects once a change enters the codebase. The four dimensions make sure that everything that should be looked at is looked at, with nothing dropped.

At ANCK, the focus of workflow customization is backport patch Code Review. Patch backporting is a high-frequency operation in maintaining a downstream kernel community: when an upstream patch is backported downstream it usually has to be adapted to the downstream context, and one has both to judge accurately whether the adaptation is correct and to confirm whether the backport is complete — whether it also brought back the later upstream fixes along with it. Around this scenario Kitta customized two kernel-specific review perspectives:

Patch semantic consistency check — identify the differences between a backported patch and its upstream original, and then judge, on the basis of semantics, whether the functionality of the two is consistent. The approach first traces back at the micro level and compares patch-level differences, then evaluates overall functional consistency at the macro level, so as to avoid missing cases where "every line is right, but the whole has changed."
Patch completeness check — intelligently trace the upstream dependency chain of the backported patch, automatically identify the corresponding upstream Fixes patches, and run a four-stage process for searching out missing Fixes, so as to rule out any regression of known vulnerabilities.

In addition, because in a downstream productization process "a quality review of the code itself is only one of the steps," Kitta also covers process-type review items such as checking PR description conventions and checking consistency between the description and the code, extending the scope of review from the code itself to the complete context of the development process.

Customizing domain knowledge, so that AI sees deeply: this decides how a problem should be judged. It relies on a hierarchical, multi-source knowledge base, splits the domain into sub-domains, and sediments typical defect patterns, so that the Agent's judgments match the real experience of experts in that field rather than resting on generic "best practices." At ANCK, this wheel landed as three things:

Data engineering for the kernel domain — collect kernel domain knowledge from multiple sources, put it through pipelines of data filtering, data synthesis, and data balancing, and obtain the domain knowledge base and the Benchmark data;
Subsystem-specific knowledge bases — the review focus differs a great deal from one kernel subsystem to another, so dedicated knowledge bases are organized by subsystem and reviews follow the actual conventions of each sub-domain;
A deep review workflow — introduce a process of "presumption of guilt — full-stack call chain analysis — self-debate": first assume the code has a problem, trace the semantics along the full-stack call chain, then verify the conclusion through self-debate, which deepens the identification of the common critical code risks in the kernel.

To make the effect of customization measurable, Kitta also built a Benchmark evaluation system: data is collected from three sources — LKML (the review process of real community Reviewers, with an even distribution of issue types and high authenticity), the sashiko benchmark (one commit maps precisely to one issue), and OpenAnolis Gitee community data — and a number of state-of-the-art code review approaches, including sashiko, review-prompts, and qoder, are brought into the comparison, with the comparative results used to guide the evolution of Skills.

▍ 3.3 Step Three: Coordinated Online and Offline Iterative Evolution

Domain knowledge is alive, and the Agent has to keep evolving too. Kitta uses a mechanism of coordinated online and offline iteration:

On the offline side, cyclic optimization with validation samples — run isolated evaluations with the validation samples, reflect and evolve based on the results, and keep calibrating the Agent's judgment criteria.

On the online side, continuous calibration from production observation — based on the content of the review comments and developers' subsequent code behavior, an evaluation framework was developed that intelligently analyzes and summarizes how effective each kind of comment turned out to be. Combined with continuous monitoring of the review acceptance rate, knowledge updates, iterative calibration, and threshold adjustment, this makes the Agent more accurate the more it is used in real conditions.

Kitta Deployment Results

▍ 4.1 Practice in the Kernel Repository of the OpenAnolis Community

Kitta has been running continuously in the real development pipeline of the OpenAnolis community, reviewing every kernel PR along a complete five-stage pipeline. Once a developer submits a PR or types /check-code-review in a comment, the review task is triggered; the whole flow proceeds step by step — first classify, then check standards and dependencies, and finally come down to code quality.

Stage 1. PR classification. Kitta first decides whether this is a purely self-developed patch or a backported patch: if the Commit contains an upstream patch declaration such as commit xxxx upstream, it is classified as a backport. The classification then determines the scope of the checks that follow — a purely self-developed patch will not run the patch completeness check or the backport consistency check, which avoids unnecessary overhead.

Stage 2. PR/Commit Message compliance check. Based on the OpenAnolis community's specification documents, Kitta checks the PR description, the Commit Title, the Commit Message, and related content. This step looks "light," but it stops a great deal of maintenance cost up front that would otherwise be caused by unclear descriptions and non-compliant titles. If there is no problem, nothing is reported; if there is one, it gives Message specification suggestions for the developer's reference.

Stage 3. Patch completeness check. For backported patches, Kitta intelligently traces their upstream dependency chain, automatically identifies the corresponding upstream Fixes patches, and searches for omissions through the flow "find the upstream Fixes → check whether they are in the PR → check whether they already exist in the downstream repository." The result falls into three categories: no missing Fixes means nothing needs to be added; merged Fixes patches means the relevant fix already exists in the downstream repository; and missing Fixes patches requires the developer to supply them, so that known vulnerabilities are not reintroduced downstream.

Stage 4. Patch backport consistency check. For a backported patch that declares an upstream source, Kitta first retrieves the original upstream patch, then compares it on three levels: the basic comparison looks at whether the modified files and code blocks match; the contextual comparison looks at whether the referenced symbols, functions, and variable definitions match and whether the adaptations made are reasonable; and the semantic comparison uses an LLM to judge whether the upstream and downstream patches implement the same functionality and semantics. Results are output in layers from consistent to text-inconsistent, semantic-inconsistent, and inconsistent, so that developers and Maintainers can quickly identify the nature of the difference.

Stage 5. Code quality review. Finally it comes down to the code itself, and Kitta classifies the issues it finds by severity as Critical or Medium / Low. Critical issues are usually attached as inline comments to the specific lines of code, and the developer either has to address them or to state explicitly that they are false positives. If no risk is found, Kitta returns LGTM.

Since Kitta went online, its core operating data is as follows:

Metric	Value
Review volume	70,000+
Gatekeeping accuracy	97%
Code merge rate	62%

Code changes in ANCK consist mainly of backported patches and self-developed patches: they must keep consistency with upstream semantics while also meeting downstream productization requirements, so the demands on review depth and accuracy are high. Running a retrospective analysis, with the online evaluation framework, over all merged PRs that Kitta had handled since going online, both core scenarios produced quantifiable results:

Backported patch scenario: acceptance rate of consistency-check comments 70.4%, accuracy 99.8%;
Self-developed PR scenario: acceptance rate of Code Review comments 41.8%, accuracy 73.6%.

Kitta's involvement means a large number of code problems are identified before they reach human review, with suggestions that can be located and verified. In practice, after Kitta gives its comments on a PR, developers often add the dependent patches or adjust the implementation directly on the strength of them, which greatly reduces the rounds of repeated questioning and back-and-forth confirmation by Maintainers. For community Maintainers this means they can move their attention from "reading every changed line" to "ruling on the key points of contention"; for developers it means faster feedback and a more predictable merge cadence.

▍ 4.2 The Benchmark Evaluation System: Making the Effect of Customization Measurable

To make the effect of customization measurable, Kitta built a Benchmark evaluation system that collects data from three sources: LKML (the review process of real community Reviewers, with an even distribution of issue types and high authenticity), the sashiko benchmark (one commit maps precisely to one issue), and OpenAnolis community data. Specifically:

LKML data lets Kitta align itself with the judgment standards of real community Reviewers;
the sashiko benchmark provides a strict one-commit-to-one-issue mapping, which keeps the direction of capability improvement tightly focused;
OpenAnolis community data brings the evaluation back to the context of the repositories Kitta actually serves.

By comparing side by side with state-of-the-art solutions such as sashiko, review-prompts, and qoder, Kitta can see clearly where it falls short on particular defect types or particular subsystems, and turn those gaps into priorities for Skill iteration. Going forward, this Benchmark system will also support plugging in other Skills/Benchmarks for evaluation, so that capability keeps being iterated and improved.

▍ 4.3 General Practice Across Domains

Kitta's capability is not limited to the kernel. By abstracting expert assets and review workflows into transferable Skills, Kitta has completed validation in six domains: operating-system kernels, compilers, file systems, network systems, cloud-native platforms, and AI computing frameworks. What these domains have in common is large codebases, deep semantics, and a high review bar — situations where traditional general-purpose review tools often cannot reach a valuable conclusion. Kitta's performance in these scenarios is as follows:

Domain	Review quality improvement
Operating-system kernel	20.4%
Compiler	24.4%
File system	17.0%
Network system	19.8%
Cloud-native platform	31.6%
AI computing framework	21.7%

Stable improvement across domains shows that Kitta's "domain customization" methodology is general: whether it is the locking semantics of the kernel, a compiler's intermediate representation, or the configuration conventions of a cloud-native platform, expert-level review capability can be built by sedimenting expert assets and customizing the review workflow. This also lays the groundwork for extending to more foundational software projects such as LLVM, CRIU, and BaseOS.

▍ 4.4 The Data Flywheel: Data Sediments Downward, Model Capability Flows Back Upward

While serving users, Kitta keeps sedimenting data that can be used with the Qwen foundation model to form a flywheel in which "data sediments downward and capability flows back upward":

Data sedimented	Capability fed back
Expert-annotated Benchmark reference datasets	Vulnerability discovery — deeper semantic understanding and discovery of potential vulnerabilities
Agent chain-of-thought trajectory data	Instruction following — more accurate tool calls and instruction adherence
User preference alignment data	Conversation with users — review comments expressed in a way that fits user needs
Users' subsequent Coding actions	Code generation — better cross-language Coding and Debug capability

The expert annotations, chain-of-thought trajectories, user preferences, and later code changes produced during review are both the fuel for improving the review service itself and a feedback input into the more fundamental capabilities of the foundation model. More importantly, this data comes from the community running in production rather than from an artificially constructed static dataset, so every iteration brings Kitta closer to real R&D scenarios.

Summary of the Practice: Boundaries and Consensus for AI Code Review

After putting this into practice, we found that if you want to maximize the effect of AI Code Review, the first thing to settle is the boundary of trust between AI and people in the collaboration. Kitta follows three principles:

Precise grading, restrained interruption — hold the line at the gate, and leave room for suggestions. Security vulnerabilities and breakages must be blocked decisively in the CI process; that is the bottom line. Other issues are presented by level and as hints only, so as not to take too much of a developer's attention, balancing the threshold at which problems are disclosed.

Let behavior be the evidence, and refuse self-congratulation — users may stay silent, but code changes cannot lie. Do not take a developer's silent disregard as approval; do not look at what users say, look at "what changed in the code." Through confidence circuit-breaking and by tracking real code changes, review and model iteration are driven by objective changes.

Humans are the backstop, and AI steps back at the right moment. AI is the searchlight: it lights up the risks and produces the evidence. The Maintainer is the steering wheel: the final decision rests with people, who are the last line of defense.

The best AI gate is not one that replaces Maintainers, but one that lets them spend their effort only where human judgment is truly needed.

Community

Kitta: Code Review Agent for Higher-Quality Code Reviews

Background and Current State

▍ 1.1 After the Surge in Code Volume, Quality Pressure Rises Sharply

▍ 1.2 The Significance of Code Review in Agentic CI

The Deeper Challenges of AI Code Review

Technical Practice of the Kitta Code Review Agent

▍ 3.1 Step One: Managing Project-Level Expert Assets

▍ 3.2 Step Two: Dual-Engine Customization of Code Review

▍ 3.3 Step Three: Coordinated Online and Offline Iterative Evolution

Kitta Deployment Results

▍ 4.1 Practice in the Kernel Repository of the OpenAnolis Community

▍ 4.2 The Benchmark Evaluation System: Making the Effect of Customization Measurable

▍ 4.3 General Practice Across Domains

▍ 4.4 The Data Flywheel: Data Sediments Downward, Model Capability Flows Back Upward

Summary of the Practice: Boundaries and Consensus for AI Code Review

Read previous post:

Alibaba Cloud Native Community

You may also like

Comments

Alibaba Cloud Native Community

Related Products

Managed Service for Prometheus

YiDA Low-code Development Platform

Cloud-Native Applications Management Solution

Phone Number Verification Service