Alibaba's Latest AI Agent Benchmark Awarded Best Resource Paper at ACL 2026
- Alibaba opensourced an agent evaluation benchmark amidst efforts to accelerate real-world agent deployment
- Alibaba’s new ACL Best Resource Paper shows leading agents still trail far behind human experts in complex rule application
- Alibaba’s latest Qwen-based agent framework tops the benchmark

Alibaba recently unveiled HSCodeComp (Harmonized System Code Competition), the first expert-level benchmark designed to evaluate deep search agents on their ability to apply complex, layered rules, such as tariff regulations in cross-border trade. It also designed a Qwen-based agent framework, which currently tops the benchmark, as continuous efforts to accelerate real-world agent deployment.
The benchmark is now open-source and available on Hugging Face and GitHub. Its accompanying research paper, titled “HSCodeComp: A Realistic and Expert-level Benchmark for Deep Search Agents in Hierarchical Rule Application,” just won the prestigious Best Resource Paper Award at the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026) in San Diego [Add link from ACL]. The Best Resource Paper award at ACL honors top datasets, benchmarks, and software toolkits that are considered fueling new AI breakthroughs.
The paper “introduces a realistic, carefully validated benchmark for evaluating agents’ ability to apply complex hierarchical rules in a multimodal product-coding setting. The dataset is especially valuable because it is based on real e-commerce data, preserves the noisy metadata and expert workflow of the task, and uses rigorous annotation and adjudication procedures that make the labels highly credible. The benchmark offers a rich testbed in a niche domain for structural reasoning, rule following, domain grounding, and diagnostic analysis of agent failures. These are capabilities that are increasingly central to current NLP research.,” commented the ACL Area Chair, the peer review platform.
Applying multi-layered, hierarchical rules is a critical bottleneck for modern AI systems. While common in cross-border trade tariffs, these complex structural rules are also fundamental to domains like legal compliance, medical diagnosis, and tax auditing. By exposing the capability boundaries of existing AI architectures, HSCodeComp establishes a rigorous testing ground for developing reliable, professional-grade AI systems capable of operating in complex business environments.

Alibaba opensourced HSCodeComp and the research paper was awarded Best Resource Paper at ACL 2026
The benchmark challenges agents to predict the exact 10-digit Harmonized System Code (HS Code) for 632 real-life products across 32 categories using real-world e-commerce data (HS Code is used for categorizing goods in international trade and plays an important role in customs clearance). To succeed, agents must correctly interpret and apply complicated, multi-layered tariff rules from sources like the eWTP and official customs rulings database. These rules often contain ambiguous language and implicit logic, making accurate classification challenging.
Extensive experiments were conducted among 14 foundation models, 6 advanced open-source agent systems and 3 closed-source agent systems. The result reveals a significant performance gap between current state-of-the-art agents (49.4% accuracy) and human experts (95.0%), suggesting that hierarchical rule application remains a major structural bottleneck for existing agent architectures. The reasons behind the performance gap include agents’ excessive reasoning that leads to unnecessary self-correction, reasoning hallucinations, and lack of domain knowledges. In addition, studies in the paper show that inference-time scaling does not improve performance, suggesting that hierarchical rule reasoning requires a new architectural approach rather than simply increasing compute. Such findings highlight that deep search with hierarchical rule application remains a critical, unsolved challenge for current AI agent systems.

The paper reveals hierarchical rule application remains a bottleneck for agent architectures
While more work is needed before AI agents can be trusted with highly specialized professional work, Alibaba also designed a Qwen-based agent framework. The framework is tailored for digital customs clearance during cross-border trade. Test results on the HSCodeComp benchmark is encouraging: it currently ranks first with an accuracy rate of 65.0%. The release of the Benchmark and the Qwen agent framework showcase Alibaba’s continuous efforts in accelerating real-world agent deployment.
This article was originally published on Alizila written by Crystal Liu
Read previous post:
Read next post:
-
-
Alibaba Cloud Community
1,605 posts | 526 followers
Follow
Comments
-
-
Alibaba Cloud Community
1,605 posts | 526 followers
Follow
Related Products
-
QwenWork
QwenWork is dedicated to helping employees strengthen their professional competitiveness in the AI era and to enabling enterprises to improve organizational effectiveness.
Learn More -
Token Plan
Build more, spend less. One plan, every modality.
Learn More -
Alibaba Cloud Model Studio
A one-stop generative AI platform to build intelligent applications that understand your business, based on Qwen model series such as Qwen-Max and other popular models
Learn More -
Qoder
From Assisted to Autonomous — Adapting to Every Programming Need
Learn More
More Posts by Alibaba Cloud Community
- The Easiest Way to Try the Latest Models in One Place: Token Plan Starting at $3 for the First Month
- 一個平台輕鬆試用最新 AI 模型:Token Plan 首月低至 $3
- Alibaba Cloud Advances AI Adoption in Malaysia with Nationwide Youth Contest and Industry Collaborations
- Alibaba Cloud Unveils New AI Initiatives at Hong Kong Summit to Advance the City’s AI Ecosystem
- QoderWake: Digital Workers Take the Initiative — No Need to @ Every Time
- What Is Model Thinking Effort? Should You Choose Low, High, or Maximum?
- High-Risk PHP and Node.js Vulnerabilities: A Scan Costs Pennies. A Miss Could Cost a Fortune.
- Alibaba Unveils Qwen4-Max: New AI Teammate for Work, Creativity, and Discovery
- Alibaba Brings Cloud and AI to the First Youth Olympic Games in Africa
- Introducing Qoder Lite: Start creating with Qoder for $10 a month