Alibaba recently unveiled HSCodeComp (Harmonized System Code Competition), the first expert-level benchmark designed to evaluate deep search agents on their ability to apply complex, layered rules, such as tariff regulations in cross-border trade. It also designed a Qwen-based agent framework, which currently tops the benchmark, as continuous efforts to accelerate real-world agent deployment.
The benchmark is now open-source and available on Hugging Face and GitHub. Its accompanying research paper, titled “HSCodeComp: A Realistic and Expert-level Benchmark for Deep Search Agents in Hierarchical Rule Application,” just won the prestigious Best Resource Paper Award at the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026) in San Diego [Add link from ACL]. The Best Resource Paper award at ACL honors top datasets, benchmarks, and software toolkits that are considered fueling new AI breakthroughs.
The paper “introduces a realistic, carefully validated benchmark for evaluating agents’ ability to apply complex hierarchical rules in a multimodal product-coding setting. The dataset is especially valuable because it is based on real e-commerce data, preserves the noisy metadata and expert workflow of the task, and uses rigorous annotation and adjudication procedures that make the labels highly credible. The benchmark offers a rich testbed in a niche domain for structural reasoning, rule following, domain grounding, and diagnostic analysis of agent failures. These are capabilities that are increasingly central to current NLP research.,” commented the ACL Area Chair, the peer review platform.
Applying multi-layered, hierarchical rules is a critical bottleneck for modern AI systems. While common in cross-border trade tariffs, these complex structural rules are also fundamental to domains like legal compliance, medical diagnosis, and tax auditing. By exposing the capability boundaries of existing AI architectures, HSCodeComp establishes a rigorous testing ground for developing reliable, professional-grade AI systems capable of operating in complex business environments.
Alibaba opensourced HSCodeComp and the research paper was awarded Best Resource Paper at ACL 2026
The benchmark challenges agents to predict the exact 10-digit Harmonized System Code (HS Code) for 632 real-life products across 32 categories using real-world e-commerce data (HS Code is used for categorizing goods in international trade and plays an important role in customs clearance). To succeed, agents must correctly interpret and apply complicated, multi-layered tariff rules from sources like the eWTP and official customs rulings database. These rules often contain ambiguous language and implicit logic, making accurate classification challenging.
Extensive experiments were conducted among 14 foundation models, 6 advanced open-source agent systems and 3 closed-source agent systems. The result reveals a significant performance gap between current state-of-the-art agents (49.4% accuracy) and human experts (95.0%), suggesting that hierarchical rule application remains a major structural bottleneck for existing agent architectures. The reasons behind the performance gap include agents’ excessive reasoning that leads to unnecessary self-correction, reasoning hallucinations, and lack of domain knowledges. In addition, studies in the paper show that inference-time scaling does not improve performance, suggesting that hierarchical rule reasoning requires a new architectural approach rather than simply increasing compute. Such findings highlight that deep search with hierarchical rule application remains a critical, unsolved challenge for current AI agent systems.
The paper reveals hierarchical rule application remains a bottleneck for agent architectures
While more work is needed before AI agents can be trusted with highly specialized professional work, Alibaba also designed a Qwen-based agent framework. The framework is tailored for digital customs clearance during cross-border trade. Test results on the HSCodeComp benchmark is encouraging: it currently ranks first with an accuracy rate of 65.0%. The release of the Benchmark and the Qwen agent framework showcase Alibaba’s continuous efforts in accelerating real-world agent deployment.
This article was originally published on Alizila written by Crystal Liu
1,454 posts | 503 followers
FollowAlibaba Cloud Community - February 9, 2026
Alibaba Cloud Native Community - April 23, 2026
CloudSecurity - June 15, 2026
CloudSecurity - March 16, 2026
Alibaba Cloud Native Community - June 8, 2026
Alibaba Cloud Native Community - May 26, 2026
1,454 posts | 503 followers
Follow
Alibaba Cloud Model Studio
A one-stop generative AI platform to build intelligent applications that understand your business, based on Qwen model series such as Qwen-Max and other popular models
Learn More
AgentBay
Multimodal cloud-based operating environment and expert agent platform, supporting automation and remote control across browsers, desktops, mobile devices, and code.
Learn More
Qwen
Full-range, open-source, multimodal, and multi-functional
Learn More
Alibaba Cloud for Generative AI
Accelerate innovation with generative AI to create new business success
Learn MoreMore Posts by Alibaba Cloud Community
