High availability for cloud stateful messaging systems is constrained by a hard engineering trade-off. Cost, steady-state performance, and failover speed are hard to get all at once. Application-layer replication shortens recovery but adds resources and operational complexity. A single-replica deployment keeps the cost and performance advantages, but on Kubernetes its recovery path is easily dragged into long tails measured in minutes by cloud disk detach/attach, scheduling, and mount flows.
The Alibaba Cloud messaging team has developed and operated Apache RocketMQ for years. Drawing on real incidents, disaster-recovery drills, and large-scale operations, we redesigned the takeover path for single-writer stateful services. In normal operation we keep the single replica, the single-node file system, and cloud block storage. On failure we use Multi-Attach and NVMe Persistent Reservation (NVMe PR below) to enforce protocol-level fencing, achieving takeover in seconds without adding any business-data replica at the RocketMQ service layer.
This work has been accepted to the FSE 2026 Industry Papers Track.
When one architecture carries thousands of tenant-isolated clusters, high availability stops being a single-instance problem. It becomes a delivery problem at scale.
Individually these clusters are small, but there are many of them; failures are scattered and cost sensitivity is high. Beyond node crashes, we must handle I/O hangs, kubelet anomalies, network partitions, and partial failures where “the process is alive but persistence has stopped advancing.” Average recovery time alone is not enough. Critical path, recovery tail, state observability, and drill repeatability all affect the overall SLO and on-call cost.
We summarize the problem as a practical “triangle trade-off”:
• Low cost: reserve no extra physical replica at the service layer, reducing redundant resources and write amplification.
• High steady-state performance: keep mature I/O optimizations such as the local file system, page cache, and batching; this also involves compatibility with community storage features.
• Fast and controllable recovery: keep an end-to-end takeover chain that is as short as possible, with unpredictable control-plane tails eliminated wherever we can.
Existing approaches can reliably meet two of the three, but rarely all three at once.
| Approach | Main advantage | Main cost |
|---|---|---|
| Application-layer replication with leader election | Fast switch after failure | Extra compute and storage resources, plus network bandwidth overhead, write amplification in the replication chain, tail latency, and the operational complexity of keeping replicas consistent |
| Kubernetes single replica + PV migration | Lower cost; keeps local file system performance | Recovery depends on eviction, scheduling, and detach/attach/mount retries; tails measured in minutes are common under I/O hangs and network anomalies |
| Distributed file system or shared storage | Fewer cloud disk migration steps | Shared I/O coordination may hurt steady-state performance; migration of existing systems and long-term operational cost are high |
Cloud block storage itself usually already provides storage-layer redundancy. For many small and medium clusters, if the service layer also keeps a full business replica for failover, resource amplification becomes significant. So our question is not “how to build a faster replication protocol” but: can we rebuild only the failover path while leaving the normal data path untouched?
Our ground rule: in normal operation the Broker still writes CommitLog, ConsumeQueue, and metadata to a single-node file system sitting on cloud block storage. We do not require rewriting the CommitLog/WAL, do not move normal reads and writes to a shared WAL or object storage, and do not introduce extra business-data replication at the RocketMQ service layer.
This yields five key design points:
To make this easier to follow, let us first clarify two key capabilities.
• Multi-Attach (multiple attachment) lets one cloud disk attach to multiple ECS instances, so old and new nodes see the same data at any given time. This is the precondition for takeover in seconds. But it solves visibility only; it does not guarantee single-writer. See Attach a single cloud disk to multiple ECS instances with Multi-Attach.
• NVMe Persistent Reservation (NVMe PR) is a distributed lock mechanism defined by the NVMe protocol, inherited from SCSI Persistent Reservation. Hosts use nvme pr commands to register, acquire, preempt, and release. The device accepts writes only from the reservation holder; other hosts’ writes are rejected at the storage protocol layer. During takeover, the incoming node preempts the reservation before coming online; this is what makes protocol-level fencing possible. For Kubernetes scenarios see Use Multi-Attach and Reservation of NVMe cloud disks for data sharing across applications.
One clarification: Multi-Attach only ensures “both nodes see the same disk.” It does not by itself guarantee single-writer. What truly prevents split-brain is NVMe PR: at any given time only the reservation holder can write to the device. Even if the old node is still running and still sees the block device, its writes are rejected at the storage protocol layer.
The implementation has two preconditions:
The first is crash-consistent recovery. After an ungraceful crash, the application and file system must recover on their own to a consistent, usable point. RocketMQ’s CommitLog, RocksDB, and ext4 journal replay all provide this capability. It is the basis for the incoming node to bring the service up directly after takeover.
The second is storage product and path constraints. The shared cloud disk must support Multi-Attach and present itself to hosts as an NVMe Reservation, so nvme pr commands can lock at the storage protocol layer and guarantee a single writer at any given time. Without either capability, protocol-level fencing cannot hold.
Figure 1, in sequence: normal operation, takeover after Node A fails, and failback after Node A recovers.
Production deployment uses a two-node pairing mode. Two nodes carry two shared disks and two Broker instances. Normally Node A runs Master Broker A and Shadow Broker B, while Node B runs Master Broker B and Shadow Broker A. Both disks are exposed to both nodes as raw block devices in advance, but only the PR holder can mount and write the corresponding data area. This keeps resource utilization high under normal operation and avoids creating Pods, rescheduling, and waiting for volume migration during failures.
The Shadow Broker here is not the same as a traditional Slave Broker. A Slave Broker belongs to the application-layer replication chain. It continuously syncs business data such as CommitLog from the Master and maintains its own replica. After a failure, leader election or the control plane promotes that existing replica to Master, so steady state continuously consumes replica compute, storage, and network bandwidth. A Shadow Broker replicates no business data: in steady state it mounts no data area and serves no business traffic; it only reads the persisted lease in the probe area and retains a lightweight takeover capability. After a failure, the Shadow does not promote a local replica. It acquires write permission on the original shared disk through NVMe PR, completes cache purge, file system mount, and crash recovery, then comes online as Master. In short, a Slave “replicates data first, then switches”; a Shadow “probes in normal operation and takes over the original disk safey on failure.” Therefore the Shadow’s existence does not mean an extra business-data replica is maintained at the RocketMQ service layer.
The system has four kinds of components, with responsibility boundaries kept as narrow as possible:
• Application: in this article, the RocketMQ Broker. It accesses only already-mounted directories and keeps its original CommitLog, index, cache, and crash-recovery logic. It does not perform PR, mount, or cache purge directly.
• Shared Disk: the sole carrier of persisted state, logically split into a probe area and a data area. The probe area stores lease information such as owner, version, and processing flags, and avoids host-cache interference through direct I/O and similar means. The data area uses a single-node file system such as ext4 or XFS to store business data.
• Prober: on the Master side it keeps advancing the lease; on the Shadow side it watches whether the persisted version changes and drives transitions among the four states Shadow, Promoting, Master, and Demoting. It makes decisions and orchestrates but holds no system-level device rights.
• Node Agent: runs on the node side and executes high-privilege operations such as PR queries and preemption, target-device cache purge, and mount/unmount. Its interface uses authentication, parameter whitelists, and audit logs, so privileges like SYS_ADMIN are never handed to the Broker container.
The probe area and data area can be implemented via physical partitions, logical devices, or reserved regions of the file system. The key is that the lease must reflect real persistence progress, while the business data area keeps local file system semantics.
A complete takeover splits into five phases.
The Master periodically writes an incrementing version to the lease on the shared disk. The Shadow reads that version directly from the same disk. Only when the version fails to advance for a full probe window does takeover trigger. The question here is whether the original Master can still complete its persistence writes, not whether the process exists. Cases where the node is up but I/O is stuck are therefore also covered.
Detection does not rely on absolute time comparisons between the two machines’ local clocks, and requires no centralized Controller, ZooKeeper, or etcd on the critical path. When the control plane is temporarily unavailable, nodes can still decide based on persisted facts on the shared disk.
The Node Agent first reads the current holder through a reservation report and checks the processing flag in the holder key. If the peer is in an uninterruptible phase such as journal replay, file system check, or Broker recovery, the incoming node backs off first instead of preempting again immediately. Combined with expected-holder validation, this process forms a CAS-like ownership transfer and prevents two recovery flows from interrupting each other.
Before acquiring the PR, the incoming node first purges any cache of the target block device that may remain locally. It then performs acquire or preempt to atomically transfer the reservation to itself. After the operation, the Agent queries the holder again; only once it confirms it holds write permission does the flow continue.
After ownership is confirmed, the target device cache must be purged again. These two purges close two distinct risk windows: the first keeps stale cache left before takeover from entering later recovery; the second clears cache that may be refilled during preemption. Otherwise, even with fencing completed at the device side, the host may still read old data from stale cache, or even corrupt the new master’s recovered state when writing back later.
Only after PR ownership confirmation and the second cache purge may the incoming node mount the data area. The file system first returns to a crash-consistent state through journal replay and similar mechanisms. RocketMQ then rebuilds a servable state with its existing CommitLog/WAL recovery flow. Recovery time is a cost tied to application and data scale; it can be reduced further by cutting unnecessary scans, tuning recovery concurrency, and shortening the recovery path.
The Broker is promoted to Master and serves traffic only after recovery completes and the readiness check passes. If any preceding step fails, the state machine never bypasses the safety boundary to come online directly. When the old node recovers and a preferred master is configured, failback goes through the same “stop service, unmount, re-preempt the lock, purge cache, mount, and recover” flow, rather than relying on the control plane to force-move the disk.
Figure 2: Shadow probes; Promoting completes PR acquisition, mount, and recovery; Master runs health checks and renews the lease; Demoting stops service and unmounts the device.
The value of an explicit state machine is separating “can see the disk,” “holds write permission,” “has finished recovery,” and “is safe to serve” into distinct phases. Shadow mounts no data area and serves no traffic. Promoting exclusively owns the transfer and recovery process. Master holds write permission and renews the lease. Demoting stops service first, then unmounts the device and returns to Shadow.
We constrain the whole flow with three safety invariants:
• I1: at most one node writes persisted state at any given time. NVMe PR enforces this at the device layer, rather than relying on either side’s subjective judgment of role state.
• I2: after takeover, stale caches left by the old master must not be used. The two target-device cache purges around the PR ensure the new master recovers from current persisted state.
• I3: recovery must be exclusive. The processing flag, holder validation, and backoff protect journal replay, fsck, and WAL recovery, so ownership contention cannot recur mid-recovery.
Beyond Multi-Attach, correct NVMe PR semantics, and crash-consistent recovery mentioned earlier, the safety guarantee has one more engineering precondition: all high-privilege operations must be executed by a trusted Node Agent. This design solves single-writer takeover under node failures, I/O stalls, partial network partitions, and control-plane delays. It does not provide availability after permanent shared-disk corruption, and it does not support true multi-writer semantics.
Another class of approaches rebuilds the storage engine, introducing shared WAL, object storage, or remote tiered storage to make compute nodes as stateless as possible. Such architectures have their applicable scenarios, but for single-writer stateful services already running at scale, migration cost, performance model, and long-term compatibility all need re-evaluation.
Our approach keeps RocketMQ’s existing single-node file system and normal read/write semantics, redesigning only the failover path. Because the HA mechanism does not replace the normal storage engine, RocketMQ keeps reusing its mature CommitLog, page cache, and crash-recovery capabilities, and maximizes compatibility with the Apache RocketMQ community edition. Cold-read flow control, compression, encryption, and future storage features in principle still follow the original data path, with no need to reimplement them for the HA mechanism.
What sets this choice apart is not “zero modification” but confining changes to clear boundaries. The Broker owns its original reads/writes and recovery; the Prober owns role decisions; the Node Agent owns device operations; and the cloud disk protocol owns final write mutual exclusion. For existing systems, this markedly reduces how deeply the Broker’s core code must be modified, along with community-upgrade cost and long-term maintenance burden.
The paper's evaluation uses Apache RocketMQ 5.3.4 with four comparison groups. S1 is traditional Master & Slave application-layer replication; S2 is Kubernetes single replica + PV migration. S3 uses a distributed file system or shared storage; S4 is this article’s Master + Shadow with Multi-Attach and protocol-level fencing. They represent the four typical approaches: application replication, PV migration, shared storage, and protocol-level takeover.
Paper Table 1: recovery time of different HA paths under node-level failures.
Results show that under node failures, S2’s recovery is stretched to minutes by the Kubernetes scheduling and cloud disk migration chain. Inspecting Kubernetes events, we found the time went mainly to two things. One is uncertainty from eviction, scheduling, and resource constraints. The other is detach, attach, and FailedMount retries: while the old node still occupies the cloud disk from the control plane’s view, the incoming node cannot take over immediately. S4 brings recovery down to seconds. This improvement is not simply speeding up some command; it removes traditional detach/attach from the critical path and keeps only failure detection, protocol-level lock preemption, and application recovery.
The paper summarizes end-to-end recovery time as:
RTO ≈ N + M
Here N is the failure-detection window determined by the probe period and the number of consecutive non-advancing rounds. M is the total time of subsequent protocol takeover and recovery, including PR acquisition or preemption, ownership confirmation, cache handling, mount, file system recovery, and Broker recovery. Traditional detach/attach no longer sits on the critical path.
M is not a fixed constant: protocol takeover and mount are usually short, while Broker recovery depends on log scale and recovery implementation and can be optimized further.
Paper Table 2: steady-state throughput and latency under a 1KB message send-only workload.
For steady-state performance, S4’s throughput is close overall to the native single-replica baseline. Shadow’s resource consumption is also low, because it replicates no business traffic and only performs lightweight lease probing. In other words, takeover in seconds does not come at the cost of extra business-data replicas and continuous write amplification.
Across different probe windows and failures such as node crashes, network anomalies, and I/O hangs, measured recovery time stays close to N + M overall: N determines the failure-detection window, and M covers subsequent protocol takeover and Broker recovery. The minutes-long tail from detach/attach does not reappear in the results. This architecture is already in production on Alibaba Cloud’s RocketMQ service and has been through tens of thousands of disaster-recovery drills. Continuous drills matter not only to prove “it can switch” but to verify that every state transition, permission operation, and fallback path can be repeatedly executed in a real environment.
The “no replication needed” here specifically means the RocketMQ service layer reserves no extra business-data replica for takeover. It does not mean the underlying cloud block storage lacks reliability redundancy. Protocol-level fencing solves who may keep writing during a failure; underlying data durability remains the responsibility of the cloud disk’s own reliability mechanisms.
This design first applies to systems with the following traits: single-writer, state kept on Multi-Attach-capable block storage, and able to recover from a crash-consistent file system and WAL. Their owners usually also want to keep local file system performance and the existing storage engine as much as possible. Beyond messaging, the same idea can apply to some databases, search, and log services, but device protocol, cache semantics, and application recovery capability must be verified item by item; one cannot merely port the “preempt the lock” command over.
More generally, this practice yields three lessons:
No single HA approach fits every scenario. For systems like Apache RocketMQ—mature local storage engine, predominantly single-writer, needing delivery at scale—we chose to keep the normal data path. Failover is rebuilt as a short path driven by persisted leases, enforced by NVMe PR fencing, and executed by node-side Agents.
The final effect: no extra business-data replica at the RocketMQ service layer, no sacrifice of steady-state performance on the single-node file system, and end-to-end RTO no longer dominated by traditional volume migration. This does not eliminate all HA costs; it confines them to boundaries you can explain, verify, and maintain.
The related paper Replication-Free Failover: Protocol-Fenced Takeover for Stateful Services has been accepted to the FSE 2026 Industry Papers Track.
Handling Checkout Peaks with Confidence: How Lemeng Built AIOps for Retail Chains
784 posts | 60 followers
FollowAlibaba Cloud Native Community - September 18, 2026
Marketplace - February 21, 2019
Alibaba Cloud Native - July 18, 2024
Alibaba Cloud Native Community - October 26, 2023
Alibaba Cloud Native Community - December 17, 2025
Alibaba Cloud Community - April 24, 2022
784 posts | 60 followers
Follow
Application High Availability Service
Application High Available Service is a SaaS-based service that helps you improve the availability of your applications.
Learn More
ApsaraMQ for RocketMQ
ApsaraMQ for RocketMQ is a distributed message queue service that supports reliable message-based asynchronous communication among microservices, distributed systems, and serverless applications.
Learn More
Container Service for Kubernetes
Alibaba Cloud Container Service for Kubernetes is a fully managed cloud container management service that supports native Kubernetes and integrates with other Alibaba Cloud products.
Learn MoreMore Posts by Alibaba Cloud Native Community
