With the global rise of Generative AI (AIGC), Large Language Models (LLMs) are transforming industries and reshaping application development. This model- and algorithm-driven revolution brings extraordinary opportunities and equally significant engineering challenges for developers building AI applications:
● How to maintain continuity for long-running, stateful conversations?
● How to allocate limited compute resources fairly and efficiently?
● How to avoid cascading blockages in multi-agent systems or complex workflows?
At the core, we need a reliable, efficient asynchronous communication mechanism to coordinate applications, data, and models. As a foundational component in distributed systems, Apache RocketMQ has excelled in microservices decoupling and data stream processing. In the AI era, addressing more complex scenarios and higher demands for performance and user experience has become a key focus of Apache RocketMQ’s evolution.
While traditional message queues have proven reliable for asynchronous decoupling, traffic shaping, and data streaming in conventional architectures, they struggle to address AI-specific demands due to fundamental shifts in interaction patterns, resource profiles, and application architectures.
Conventional applications rely on stateless, millisecond-level request-response patterns. AI interactions (e.g., multi-turn or multimodal sessions) are different: a single inference can take seconds to minutes, rely on many turns of context, and consume significant compute resources. If an AI app uses long‑lived HTTP connections or WebSockets with synchronous, blocking backends, intermittent issues like network jitter, gateway restarts, or timeouts can drop context and interrupt inference—wasting compute resources and degrading user experience.
AI inference depends on expensive GPUs. Spiky traffic can overwhelm model services and waste capacity. While traditional MQs can buffer bursts, they often lack fine-grained consumption control in multi-tenant shared pools, making differentiated and efficient scheduling difficult and lowering utilization.
AI agents and multi-step workflows are long-running, coordinated tasks. With synchronous calls, a blockage at a single node can cascade into end-to-end failure. What’s needed is a reliable, efficient asynchronous hub to connect independent, long-lived agents or workflow nodes, enabling non-blocking coordination so that distributed intelligent systems stay stable.
Furthermore, traditional MQs face other challenges of AI scenarios: large multimodal payloads often exceed strict message size limits, forcing awkward workarounds that increase complexity and risk; and topic administration is frequently manual or script-heavy, leading to operational overhead and potential resource leaks.
Since version 5.0, Apache RocketMQ has undergone a comprehensive upgrade to a cloud-native architecture, with a complete restructuring across client and server components: compute-storage separation for elastic scaling, multi-replica storage for high availability, lightweight SDKs for flexible clients, and more. The result—high elasticity, high availability, and lower costs—has laid the groundwork for addressing AI-era engineering challenges.
To face the new challenges in AI scenarios, Apache RocketMQ has undergone a strategic upgrade, evolving from traditional messaging middleware into an AI message engine, positioning it as a critical infrastructure for next-generation AI applications.
Two core, disruptive innovations drive this evolution:
Dynamically create millions of Lite-Topics, ideal for long-running sessions, AI workflows, and agent-to-agent (A2A) interactions. This dramatically increases scalability and flexibility to match AI scenarios’ complex communication patterns.
With burst traffic smoothing, rate-limited consumption, adaptive load balancing, and priority queues, Apache RocketMQ enables fine-grained control over scarce compute resources, ensuring efficient utilization under high concurrency and multi-tenancy.
These innovations break through traditional MQ limitations and map precisely to AI workloads’ unique needs, providing a stable and efficient messaging backbone for modern AI systems.
AI interactions are long-running, multi-turn, and resource-intensive. If an app relies on long-polling protocols like Server-Sent Events (SSE) or WebSockets, any connection interruption (due to a gateway restart, timeout, or network instability) risks losing session context or terminating in-flight inference, wasting valuable compute resources. A robust session management mechanism is critical: it should preserve context for long sessions, reduce retries, and waste compute resources, and keep application code simple.
To solve this, RocketMQ for AI introduces a lightweight, transformative pattern: Session-as-Topic. The system dynamically creates a dedicated Lite-Topic for each independent session or question.
When a client starts a session with an AI service, the system creates a topic named after the session ID (e.g., chatbot/{sessionID} or chatbot/{questionID}). All interaction history and intermediate results for that session are transmitted as messages through this topic in order. If the client disconnects, it simply resubscribes to the original Lite-Topic (e.g., chatbot/{sessionID}) upon reconnection to resume from the last known point and receive subsequent results.
This pattern reconciles stateless backends with stateful UX. It eliminates the need for session state persistence, reconnection handling, and data consistency checks, reducing engineering complexity and avoiding wasted compute resources from restarts. Users get a smooth, continuous, and stable AI experience.
This innovative pattern is made possible by RocketMQ's powerful features designed for AI scenarios:
● Scalability for Millions of Topics: Efficiently manage millions of Lite-Topics in a single cluster, providing isolated topics for massive concurrent sessions without performance degradation.
● Lightweight Resource Management: Topic creation and teardown are lightweight and automated. RocketMQ can auto-create and reclaim Lite-Topics on demand (e.g., when clients disconnect or TTL expires), preventing leaks and reducing operational overhead.
● Large Payload Support: Handle message payloads of tens of MB and beyond, accommodating typical AIGC loads like long prompts, high-resolution images, and lengthy documents.
● Ordered Messaging: AI UIs often stream tokens to reduce latency. RocketMQ’s ordered messages ensure outputs are delivered in order for a coherent session experience.
● Full Observability: RocketMQ natively supports OpenTelemetry metrics and tracing, allowing real-time monitoring of message send/receive rates and backlogs. It also enables querying of detailed message traces for debugging and optimizing multi-agent systems.
Alibaba’s Security team built an assistant that faced context loss and task interruption under high concurrency. By adopting Lite-Topics for session persistence, the system now automatically retains and recovers session state across multi-turn interactions, simplifies engineering, reduces wasted compute resources from interruptions, and boosts user experience and operational efficiency. Multiple Alibaba Cloud products have also upgraded their AI-powered Q&A bots with this pattern, validating its generality and effectiveness.
LLM services generally face two core challenges in resource scheduling:
● Load Mismatch: Frontend requests are often bursty, while backend compute resources are limited and relatively stable. A direct connection can easily lead to service overload and crashes or wasted compute resources.
● Undifferentiated Allocation: After smoothing traffic, ensuring that high-priority tasks receive preferential access to valuable compute resources becomes key to maximizing overall service value.
Apache RocketMQ acts as a buffer layer between frontend requests and backend computing services, "shaping" irregular bursts into a stable and controllable request stream. With rate-limited consumption and priority queues, it becomes a controllable "compute scheduling hub" for fine-grained request control, improving utilization and service quality.
A series of core features in RocketMQ provides a solid foundation for intelligent compute scheduling:
● Traffic Shaping: RocketMQ buffers bursty requests, allowing LLM services to adaptively balance the load based on their processing capacity, similar to a sliding window pattern, avoiding system overloads and resource waste.
● Rate-Limited Consumption: Set quotas at the consumer group level. Set per‑second call limits for LLM services to maximize throughput while ensuring core resources are not overloaded.
● Priority Queues:
Alibaba Cloud Model Studio uses RocketMQ to smooth bursts at the gateway, turning irregular inbound load into stable downstream traffic. Combined with message priority, it prevents high-volume tenants from starving low-volume tenants, increasing utilization and fairness.
Alibaba Cloud AI Coding Assistant Lingma upgraded its codebase RAG pipeline from synchronous to asynchronous with RocketMQ, enabling vectorization at scale and traffic smoothing to stabilize end-to-end performance and reliability.
Google’s A2A (Agent-to-Agent) protocol advocates asynchronous communication to avoid blocking in long-running AI tasks. It decouples a single request–reply into an initial request and an asynchronous notification (pushNotificationConfig). In agentic workflows, each node must notify downstream nodes when it completes—a perfect fit for asynchronous messaging.
Because AI tasks are long-running, workflows face the same "sync calls cause cascading blocks" problem. Whether across agents (external A2A) or within a workflow (internal task handoffs), the scalable solution is to convert long-lived, stateful interactions into stateless, event-driven, reliable asynchronous notifications.
RocketMQ’s Lite-Topics make this request–reply pattern straightforward at scale:
● Dynamically Create a Reply Channel: Agent A sends a request to Agent B without waiting. The request contains a unique reply topic (e.g., a2a‑topic/{taskID}). Agent A subscribes to that topic; RocketMQ auto-creates the lightweight reply topic on first use—an exclusive async channel for this task.
● Deliver Results Asynchronously: Agent B processes at its own pace and publishes the result to the specified reply topic (a2a‑topic/{taskID}).
● Auto-Reclaim Communication Resources: After Agent A consumes and processes the result, it unsubscribes. RocketMQ detects that the topic has no consumers and automatically cleans it up after the configured TTL. No manual cleanup and no leaks.
The strength of Lite-Topics lies in system-level design: millions of concurrent topics, created on demand and reclaimed automatically, address scalability and operability challenges for large-scale agent orchestration. Ordered delivery preserves logical correctness for streaming or multi-step tasks. Built-in persistence and HA ensure eventual consistency and reliability. Together, these make RocketMQ a robust, efficient, and scalable async foundation for A2A.
Alibaba AI Labs has built an efficient and reliable agent orchestration system based on RocketMQ for multi-agent workflows. Each node is event-driven with durable messaging. Lite-Topics enable fine-grained, point-to-point communication between agents. Even if agents restart or calls time out, durable event streams support reliable retries to continue partially executed tasks—reducing waste and improving user experience.
Supporting millions of Lite-Topics per cluster required deep architectural changes. Two bottlenecks in the legacy design had to be addressed: file-based indexing and metadata management could not scale to this cardinality, and long-polling notification struggled with latency and concurrency when a single consumer subscribed to very large numbers of topics.
Two hard problems needed solutions:
● Scalable metadata storage and indexing for millions of Lite-Topics
● Highly efficient message distribution and delivery for massive subscription sets
Maintaining file-backed indexes per topic at this scale would be operationally prohibitive. Apache RocketMQ redesigned metadata and indexing with its LMQ storage engine and KV store:
● Unified Storage with Multi-Path Dispatch: Messages are stored once in the CommitLog. A multi-dispatch mechanism creates per-topic consumption indexes (ConsumerQueue, CQ) for different Lite-Topics.
● Upgraded Index Engine: RocketMQ replaces traditional file-based CQ with a high-performance KV engine (RocksDB) and stores queue index and physical offsets as key-value pairs, leveraging RocksDB’s excellent sequential write performance to manage millions of queues efficiently.
On top of the Lite-Topic storage model, RocketMQ reworked message distribution and delivery mechanism—introducing an innovative "Event-Driven Pull" approach for consumers that subscribe to tens of thousands of topics:
● Subscription Set Management: Brokers manage each consumer’s subscribed Lite-Topic set and support incremental updates, allowing real-time, proactive monitoring of the matching status between messages and subscriptions.
● Event-Driven Ready Set: When new messages arrive, the broker matches them against the Subscription Set and appends qualifying messages (or their indexes) to a per-consumer Ready Set.
● Efficient Ready-Set Polling: Consumers poll the Ready Set to retrieve all matched messages. Brokers can coalesce and batch messages across multiple topics into a single response, dramatically reducing network round-trips and improving throughput.
With these storage and distribution upgrades, Apache RocketMQ delivers the Lite-Topic model at scale. RocksDB replaces file-based indexing for efficient metadata management at million‑topic scale, while "event-driven pull" shifts from massive client-side polling to a single, efficient pull of broker-maintained aggregation, ensuring low latency and high throughput in scenarios with massive subscriptions.
RocketMQ for AI marks the shift from traditional messaging middleware to an AI-era message engine. With disruptive innovations in lightweight communication and intelligent resource scheduling, RocketMQ extends beyond conventional MQ capabilities to become key infrastructure for highly available, scalable AI applications.
These enhancements have been validated at scale within Alibaba and across Alibaba Cloud offerings such as Alibaba Cloud Model Studio and Lingma, proving maturity and reliability in high-concurrency, complex AI scenarios.
This is just the beginning. AI engineering is evolving rapidly. As core infrastructure, Apache RocketMQ has plenty of room for continued optimization and innovation. The Alibaba Cloud Messaging team will continue to iterate and upgrade based on users' AI scenarios, collaborate with contributors in the Apache RocketMQ community to refine core AI capabilities, and progressively contribute solutions and features validated by Alibaba Group's AI business back to the open-source community.
We believe ongoing innovation and open collaboration will establish RocketMQ for AI as the standard for AI-native messaging (AI MQ), helping developers worldwide build the next generation of intelligent applications more easily and efficiently, and accelerating standardization, adoption, and ecosystem growth in AI engineering.
What Challenges Does Agent Face on the Path from Q&A to Autonomous Execution?
714 posts | 58 followers
FollowAlibaba Cloud Native Community - December 17, 2025
Alibaba Cloud Native - June 6, 2024
Apache Flink Community - November 7, 2025
Alibaba Cloud Native Community - February 1, 2023
Alibaba Clouder - June 30, 2020
Alibaba Developer - December 17, 2018
714 posts | 58 followers
Follow
ApsaraMQ for RocketMQ
ApsaraMQ for RocketMQ is a distributed message queue service that supports reliable message-based asynchronous communication among microservices, distributed systems, and serverless applications.
Learn More
Alibaba Cloud Model Studio
A one-stop generative AI platform to build intelligent applications that understand your business, based on Qwen model series such as Qwen-Max and other popular models
Learn More
Short Message Service
Short Message Service (SMS) helps enterprises worldwide build channels to reach their customers with user-friendly, efficient, and intelligent communication capabilities.
Learn More
ChatAPP
Reach global users more accurately and efficiently via IM Channel
Learn MoreMore Posts by Alibaba Cloud Native Community
