Apache RocketMQ's Evolution for AI: LiteTopic Powers Million-Scale Multi-Agent Session Collaboration

This article introduces Apache RocketMQ's LiteTopic, a lightweight messaging model that enables reliable, session-level collaboration for million-scale multi-agent AI systems.

By Li Zhou

Adapted from the Apache 2026 talk “Apache RocketMQ for AI: Reliable Collaboration for Multi-Agent Systems.”

Over the past year two AI systems went to production on RocketMQ inside Alibaba Cloud: the coding agent Qoder Cloud Agents, where a single task can run for hours or days; and the Model Studio inference gateway, which schedules requests across millions of tenants. They look very different but ask a messaging system the same thing: session state, task handoff, and failure recovery should be carried reliably by the infrastructure, not reimplemented over and over in the application layer.

This article covers the new constraints AI applications place on asynchronous collaboration, RocketMQ’s AI-facing response (LiteTopic), and the concrete problems it solved in both cases.

1. New constraints AI applications place on asynchronous collaboration

Compared with traditional applications, AI applications differ fundamentally along three dimensions: channel topology, task duration, and cost structure.

Channel topology. A traditional application fixes its business flow at design time—nodes and message flows barely change after launch, so channels and their topology can be planned up front. An AI agent’s execution path is decided at runtime: tasks are decomposed autonomously and spawned dynamically, so the number and topology of channels cannot be enumerated at design time.

Task duration. Traditional asynchronous consumption takes milliseconds to hundreds of milliseconds. AI tasks involve multiple rounds of reasoning and tool calls, so a single run often takes minutes, a long session can span days, and the duration is highly unpredictable. Turns within a task rise sharply and intermediate state is read and written frequently, making state and session management infrastructure-layer problems.

Cost structure. GPU compute costs far more per unit than CPU, so in AI any idle compute shows up directly in cost. This constraint runs through many of the design choices below.

Under these constraints, the traditional event-driven model breaks down in three ways.

First, the backlog growth rate. Entry traffic is unchanged, but per-message processing has risen from milliseconds to minutes, so queues build up at an entirely different rate. Spikes once handled by adding consumers now need multiples of the compute to drain.

Second, head-of-line blocking. Traditionally a queue is shared by a consumer group on the assumption that the queue’s messages are homogeneous and interchangeable. In AI, one message is one specific user’s one specific session; its long runtime is a property of the task, not a system fault. In a shared queue it blocks everything behind it, leaving unrelated sessions stuck waiting. The mitigation is finer granularity, and what actually works is per-session granularity—but a traditional Topic’s creation and metadata cost makes “one Topic per session” untenable.

Third, the cost of re-running inference. A single inference’s compute is already spent. If a process restart, instance reschedule, or network fault loses intermediate results, that work must be recomputed. Conversely, persisting results reliably and reading them back after recovery avoids calling the model again. Message and event durability here is not just about reliability—it bears directly on cost.

From this, a messaging system needs three capabilities: support a vast number of dynamically created, mutually independent high-speed channels; push isolation from the Topic level down to the session level, giving each session its own ordering and fault boundary; track which session each message belongs to, providing session-level ordering and affinity. Together they point to a new Topic model.

2. The design and implementation of RocketMQ LiteTopic

RocketMQ LiteTopic’s core idea is to map each AI session to its own lightweight Topic, rather than have all sessions share one business Topic.

Session-level granularity introduces two engineering problems to solve first: how to organize the consumption-queue index when millions of channels coexist, and how to know which channels have messages to deliver under millions of subscriptions. LiteTopic answers both—at the storage layer it replaces the file-based consumption-queue index with a RocksDB KV engine, and at the delivery layer it replaces long polling with event-driven dispatch.

This is not a rewrite of the RocketMQ storage kernel. Sequential writes, zero-copy, master-slave replication, and flush policies—proven at large scale—stay unchanged; LiteTopic only adds one index layer on top and a new set of subscription and delivery semantics, which is what sets its maturity and onboarding cost.

Topic model: Parent Topic and LiteTopic, two levels

LiteTopics sit under a Parent Topic, forming a two-level structure.

The Parent Topic is the namespace and control boundary. It must be created in advance, and controls such as permissions, routing, and quotas attach at this level. Parent Topics are few—usually one per business domain or Worker pool.

A LiteTopic is a runtime dynamic unit with three properties: ordered, isolated, replayable. It needs no pre-creation—created automatically when the first message arrives, reclaimed by TTL once idle. The application only tags each message with its session.

Under this model the cost of creating and destroying a logical channel drops low enough to allocate one per session, channel lifecycle aligns with session lifecycle, and operations teams never intervene as channels are added or removed.

Consumption model: subscription granularity drops from Group to Client

The new Topic model requires the consumption model to follow. RocketMQ’s original clustered consumption was organized by Group: every instance subscribed to the same set of Topics and load-balanced within the group. This suits homogeneous tasks but cannot express differences between instances, so it cannot route a specific message to a specific instance.

LiteTopic refines subscription to support both the Group and single-Client levels. A message can be routed to a specific instance, and each LiteTopic is guaranteed to be consumed by exactly one Client at any moment—so one session’s messages are never split across instances, avoiding out-of-order delivery and state races. Different instances in a consumer group can also subscribe to different LiteTopic subsets, spreading load along the session dimension.

Storage structure: write once, index many ways

RocketMQ storage is layered: message bodies append sequentially to the CommitLog, and the ConsumeQueue is only a lightweight index to physical locations. LiteTopic keeps this structure and confines changes to the index layer: a background dispatch thread builds two indexes in parallel—one ConsumeQueue by Parent Topic, one LiteCQ by LiteTopic. Each message is still written once, write amplification stays bounded, and a consumer pulls exactly its channel’s messages by LiteTopic.

Index engine: RocksDB for the consumption-queue index

The original ConsumeQueue was file-based, one file per queue. Fine at tens to hundreds of Topics, but millions of LiteTopics means millions of small files—the overhead of file handles, memory-mapped regions, and metadata management grows nonlinearly, and writes degrade into a flood of random I/O.

After the change, all LiteCQ indexes converge into one KV engine: the Key is Topic + QueueID + Offset, the Value the message’s physical offset in the CommitLog. Millions of LiteTopics share one KV store, a single lookup is O(1), and metadata no longer balloons with channel count. RocksDB’s LSM structure suits high-frequency writes and huge numbers of small records—matching this workload. Moving from “a file per queue” to “shared KV, O(1) metadata” is the precondition for a million channels to coexist.

Delivery mechanism: a Ready Set instead of full polling

The storage layer handles how many channels fit; the delivery layer must locate which are active. In long polling the consumer sends periodic requests and the Broker scans the relevant queues. At a million queues these full scans drive up CPU cost, and most are wasted—at any moment only a small fraction of channels have messages.

LiteTopic introduces two structures: the Lite Subscription Set holds subscriptions; the Ready Set aggregates channels that genuinely have messages to deliver. A written message triggers an event match; messages hitting a subscription are aggregated into the Ready Set, and consumers take work only from it. This mirrors epoll’s improvement over select: scheduling cost falls from O(total subscriptions) to O(active). Existing sessions may number in the millions while active channels in a given interval are only in the thousands, so the system pays scheduling overhead only for the active part.

Application-facing capabilities: create on demand, auto-expire, differentiated subscription, consumer suspend

These mechanisms surface as four capabilities, each answering a problem raised in section 1.

• Create on demand: a channel is created when its first message arrives—no pre-requesting, no config registration—solving resource provisioning for vast numbers of channels.

• Auto-expire: idle channels are reclaimed by TTL, with no explicit teardown. Create-on-demand must be paired with auto-reclaim, or metadata from past sessions accumulates forever.

• Differentiated subscription: messages can be delivered precisely to a single Client, and instances in a consumer group can subscribe to different LiteTopic subsets, achieving session-level isolation.

• Consumer suspend: when a consumer returns Suspend(N), only the current LiteTopic pauses for N milliseconds; other channels are unaffected. It replaces sleep-based throttling in the consumer thread: while sleeping, the thread is tied up and other sessions must wait; with Suspend, the Broker records when the channel next becomes deliverable and the thread returns to the pool at once to serve other sessions. Where GPU cost dominates, this difference bears directly on utilization.

Onboarding: changes confined to session tagging and throttling

The producer adds a session-dimension identifier when building a message:

// Produce: write per session; the channel is created automatically by the Broker
Message msg = provider.newMessageBuilder()
    .setTopic(PARENT_TOPIC)
    .setLiteTopic("session-" + sessionId)
    .setBody(payload)
    .build();
producer.send(msg);

The consumer subscribes to all channels under the Parent Topic and returns Suspend when it needs to throttle:

// Consume: subscribe to all channels under the parent Topic, throttle per channel
consumer.subscribe(PARENT_TOPIC, "*");

// Suspend only the current channel, without blocking the consumer thread
return ConsumeResult.suspend(Duration.ofMillis(50));

The above is illustrative; the actual API depends on the version in use. Your existing programming model, client dependencies, and operations system stay as they are—changes are confined to session tagging and throttling.

3. In production

This section leads with Qoder Cloud Agents, which uses the capabilities above fairly completely; the Model Studio gateway, a different class of problem, comes second.

Case 1: Qoder Cloud Agents

Core difficulty: ordered handoff and failure recovery under a brain–hands split

Qoder Cloud Agents is a cloud-hosted coding-agent service: given a requirement it autonomously reads, edits, tests, and iteratively fixes code, and a single task runs long. Its architecture separates the “brain” from the “hands”—the Agent Runtime decides, the Sandbox Worker pool executes, all event-driven. Session state is not bound to a specific instance, so the system scales without dropping in-flight sessions and supports tens of thousands of concurrent inference instances.

To be clear, a distributed architecture does not itself shorten any single inference; its payoff is utilization, concurrency, and cost efficiency. The real difficulties are recoverable execution state, ordered task handoff, complex agent-loop semantics, failure recovery, and version evolution—all message-and-event-layer problems.

Built on LiteTopic: dual request/response channels, one lane per session

The Agent Runtime and Sandbox Worker communicate asynchronously both ways over two Parent Topics—one for requests, one for responses. Workers subscribe only to requests, the Runtime only to responses, so neither blocks on the other’s processing. Each session owns one lane per Parent Topic—a LiteTopic named by session ID, created with the first message and reclaimed by TTL after the session ends.

What flows through the messaging system are facts and events—task instructions, execution results, state transitions—not bulky intermediate state. Checkpoints carry progress and persist to storage; bulky objects and artifacts live on the storage side too, with only references in messages. The boundary must be explicit: the messaging system delivers events in order and reliably, while the authoritative source of a task’s progress is the Checkpoint, not inferred from message ACK timing. So instance restarts, session migrations, and event replays never distort progress.

Under this structure, messages are ordered within a session while sessions run in parallel—ordering and concurrency at once. When traffic spikes, the Broker absorbs the backlog and Workers pull at their own pace, producing channel-level backpressure: a slow session does not cascade into global overload, and fault impact stays within one channel. For isolation, Parent Topics separate Worker pools, LiteTopics separate business units, and the brain and hands scale independently.

Four complex scenarios: parallelism, interruption, webhooks, suspend-and-resume

Multi-agent parallelism: A session spawns several execution lines. Tasks are dispatched in order as LiteTopic = session-id; Workers claim tasks and run them in parallel; results are aggregated on the scheduling side via a Mailbox and a Barrier, gaining deterministic dispatch, execution concurrency, and join semantics at once.
Interruption handling: When the agent receives an added user instruction mid-execution, the message first enters its Mailbox and is injected at the next safe boundary before the current task completes; to abort a running task, a stop command goes down a separate control channel, and execution then resumes via LiteTopic. Data plane and control plane must be kept apart here.
Webhook callbacks: Organized as LiteTopic = endpoint-id, so callbacks for one endpoint stay ordered while different endpoints run in parallel—if an endpoint slows or fails, only its own channel blocks.
Suspend and resume: When the agent waits for user input, a tool’s response, or a child agent’s completion, it persists pending state and releases the Worker—no compute is held while waiting; when the event returns, the next task segment is dispatched via LiteTopic and any idle Worker picks it up. Traditionally a session is pinned to a Worker, causing three bottlenecks: a dropped connection loses context, instances idle during waits, and sessions migrate painfully on scale-out. On-demand assignment removes all three.

These four scenarios are expressed uniformly under one LiteTopic architecture, with no separate mechanism per scenario.

Case 2: Model Studio gateway

Core difficulty: the nature of rate limiting changed

A traditional gateway rate-limits by judging whether a caller’s request rate exceeds a quota and rejecting if so, on the premise that backend capacity scales elastically and limiting only stops extreme traffic from overwhelming the system. Model inference breaks that premise: GPU is the scarcest resource today, and backend throughput has a hard ceiling. Rate limiting therefore stops being a binary allow/reject decision; it must also decide at what pace to forward requests to the backend.

At million-tenant scale, plain threshold limiting amplifies two errors at once. False rejections: a request the backend could take is rejected outright because its instantaneous rate crossed the threshold—the caller gets a 503 even though the backend is not saturated. Over-admission: traffic across several dimensions peaks simultaneously, none individually over threshold, yet the sum crushes the backend. The cause is that traditional limiting has only allow and reject—no third state that accepts a request first and forwards it at a suitable pace.

Built on LiteTopic: a leaky-bucket array that throttles precisely by user + model

After the rebuild, the gateway keeps only a fixed-window hard cap to guard against extremes. Each request is tagged via setLiteTopic by user + model and written into the messaging system; the LiteTopics form a leaky-bucket array that does the actual throttling. Buckets are created on demand and reclaimed by TTL, so you never pre-provision resources for tenant-model combinations. A single consumer group consumes at a steady rate via subscribe(parent, "*"), returning Suspend(N ms) to throttle—precisely controlling one user + model combination’s output rate without affecting others.

In effect, limiting went from a single dimension to thousands, and the rejection ratio dropped to about a tenth—more valid requests admitted on the same backend compute and far fewer wrongly rejected. Throttling precision is millisecond-level: the throttle tracks the backend’s real-time load and adjusts output rate. This solution now runs on the public-cloud Model Studio gateway, serving millions of tenants.

The two cases are shaped very differently—one about ordered handoff, one about compute allocation—but they lean on the same capabilities: session-grained channels, create-on-demand with auto-reclaim, and throttling that never blocks the consumer thread.

4. Community and academic progress: merged into the Apache mainline, paper accepted to FSE 2026

Community: merged into master, OSCAR and InfoQ awards

LiteTopic has been merged into the Apache RocketMQ master branch as a mainline capability. RocketMQ-A2A is also open to the community, and an MCP transport layer is currently being explored to extend this transport to agent-to-tool communication. This work won the China Academy of Information and Communications Technology (CAICT) 2025 OSCAR “Open Source + AI” Summit Case Award and InfoQ’s AI Open-Source Star Project Award.

Academic: the RocketMQ-A2A paper accepted to FSE 2026

The paper’s claim: treat multi-agent collaboration as execution infrastructure

The design appears in the paper “RocketMQ-A2A: Reliable Session-Level Replayable Event Streams for Large-Scale Multi-Agent Collaboration,” accepted to FSE Companion 2026, held in Montreal, Canada in July 2026. The infrastructure’s responsibility goes beyond message delivery to include recovery after failure and post-hoc auditability.

The paper’s design: dual-plane architecture and reliability semantics

The paper splits the two kinds of communication into two planes. Task dispatch goes over ordinary Topics, optimized for throughput—traffic bursts become MQ backlog that Workers pull at their own capacity. Result return and auditing go over LiteTopics, per session—each with one ordered, persistent, replayable result stream. The Supervisor no longer blocks waiting on long tasks; it advances session state by events.

On reliability: on the task plane, a Worker ACKs a task only after its terminal event is successfully written to a LiteTopic; on the return-and-audit plane, the Supervisor dedupes by (sid, tid, seq) and supports replay from a given offset or seq. The system provides at-least-once delivery and replayability—Broker-confirmed tasks and events are never silently lost, but duplicates can occur, so side-effecting operations still need idempotency-key protection in the application.

The paper’s data: latency stays flat at 15 million sessions and 50k TPS

Load-test results: 5 Brokers holding 2.5 million sessions average about 21 ms and P95 about 47 ms at 12.5k TPS, and about 29 ms and P95 about 61 ms at 25k TPS. When sessions rise to 6 million, 5 Brokers hit about 84% CPU at 25k TPS, near this configuration’s saturation point. With 10 Brokers, at 6 million sessions and 50k TPS, latency averages about 26 ms with P95 about 81 ms and CPU about 47%; pushing sessions to 15 million while holding 50k TPS keeps latency about 26 ms with P95 about 71 ms. Under the fixed 10-Broker / 50k-TPS condition, scaling existing sessions 2.5× leaves latency essentially flat, and the cost shows up mainly as higher CPU—control-plane overhead is driven mostly by active-session count, only weakly by existing-session count. Concurrency in the 200k LiteTopic range also stays stable.

5. Summary: from message middleware to agent-collaboration infrastructure

LiteTopic does two things: it drops the cost of a message channel low enough to allocate per session, and it ties scheduling cost to active channels instead of total channels—making it practical to run a million channels at once. Ordering, isolation, replayability, and non-blocking throttling build on this.

For teams pushing agent systems to production, session state, task handoff, and failure recovery are better carried by the infrastructure than reimplemented over and over in the application layer. LiteTopic is already in the Apache RocketMQ mainline, ready to use; join the community to discuss where it goes next.

Community

Apache RocketMQ's Evolution for AI: LiteTopic Powers Million-Scale Multi-Agent Session Collaboration

1. New constraints AI applications place on asynchronous collaboration

2. The design and implementation of RocketMQ LiteTopic

Topic model: Parent Topic and LiteTopic, two levels

Consumption model: subscription granularity drops from Group to Client

Storage structure: write once, index many ways

Index engine: RocksDB for the consumption-queue index

Delivery mechanism: a Ready Set instead of full polling

Application-facing capabilities: create on demand, auto-expire, differentiated subscription, consumer suspend

Onboarding: changes confined to session tagging and throttling

3. In production

Case 1: Qoder Cloud Agents

Case 2: Model Studio gateway

4. Community and academic progress: merged into the Apache mainline, paper accepted to FSE 2026

5. Summary: from message middleware to agent-collaboration infrastructure

Related links:

Read previous post:

Alibaba Cloud Native Community

You may also like

Comments

Alibaba Cloud Native Community

Related Products

ApsaraMQ for RocketMQ

ApsaraMQ for MQTT

Message Queue for RabbitMQ

AgentCore