Migrate self-managed Apache RocketMQ clusters to ApsaraMQ for RocketMQ - ApsaraMQ for RocketMQ

Product comparison

Alibaba Cloud ApsaraMQ for RocketMQ offers significant advantages over open source RocketMQ in technical architecture, elasticity, O&M, and enterprise-grade features.

Item	Self-managed Apache RocketMQ cluster	ApsaraMQ for RocketMQ 5.x series
Storage elasticity	Lacks a resource pool and uses a storage-compute coupled architecture.	Leverages large-scale resource pools from the cloud infrastructure and uses a storage-compute decoupled architecture.
API and SDK access	Supports Apache RocketMQ SDKs.	Supports Apache RocketMQ SDKs. Supports Alibaba Cloud ONS SDKs.
Technical architecture	Uses local disks. Storage space cannot be elastically scaled. Insufficient space can lead to premature data cleanup. Incurs high storage costs due to multiple replicas.	Built on a large-scale cloud storage foundation and is fully serverless. You can use storage space as needed without planning for scaling. Billing is on a pay-as-you-go basis. For the same number of replicas, the cost is only one-third of that for a self-managed cluster.
Compute elasticity	Requires you to plan capacity based on cluster usage. Requires reserving capacity. Scaling in is complex. Cannot support burst traffic due to scale-out speed limitations.	Provides elasticity based on the resource pools of the cloud infrastructure. Planned elasticity: You can upgrade or downgrade specifications at any time, and the changes take effect in minutes. Unplanned elasticity: Supports burst traffic. You do not need to reserve large amounts of capacity, which saves costs.
O&M complexity	Manual command-line O&M is costly and high-risk. Lacks an observability and monitoring system.	A fully managed Platform as a Service (PaaS) that requires no O&M or deployment of machine resources. Provides out-of-the-box features such as dashboard diagnostics, message tracing, monitoring, and alerting.
Stability guarantee	Requires experienced technical staff for self-managed O&M.	Provides a clear Service-Level Agreement (SLA) guarantee: Data reliability: up to 99.99999999%. Service availability: up to 99.99%.
Enterprise-grade features	Requires experienced technical staff for custom development.	Provides out-of-the-box features such as end-to-end canary release, message routing and replication, extract, transform, and load (ETL), and event integration and analysis.
Systematic disaster recovery	Requires experienced technical staff for self-managed O&M.	Provides the following systematic disaster recovery plans: Intra-city active-active disaster recovery Geo-disaster recovery

How the migration solution works

Basic requirements

RocketMQ is widely used in core business scenarios such as order processing and online payments. The upstream and downstream systems that rely on messaging services have strict stability requirements, so you must migrate RocketMQ clusters with care. The migration solution must meet the following requirements:

No service interruptions

The migration process must not affect upper-layer messaging applications or cause a significant number of errors or failures.
No significant message duplication

The migration must not generate many duplicate messages. This prevents your business from having to handle systematic message duplication caused by the migration.
Messages are sent and received in real time.

The end-to-end message latency must not increase significantly during the migration. This prevents message delivery failures.

Solution design

To meet these migration requirements, ApsaraMQ for RocketMQ provides a migration tool that facilitates smooth and transparent switchovers. The tool handles the migration of both metadata, such as topics, groups, and consumer offsets, and business messages.

Metadata migration: Metadata is read from the source self-managed RocketMQ cluster and copied to the destination ApsaraMQ for RocketMQ cluster to create and synchronize the metadata.
Message migration: ApsaraMQ for RocketMQ uses a built-in routing control component to proxy topic routing information in the background. This component enables the dynamic switching of client read and write traffic, making the process seamless to your services.

As shown in the preceding figure, assume that Topic A in the source cluster has eight read partitions and eight write partitions.

The data migration task also creates a Topic A in the destination cluster with the same number of read and write partitions as the source cluster.
When the message migration process starts, the ApsaraMQ for RocketMQ routing control component manages the topic routing information for both the source and destination clusters. It dynamically returns read and write partition information to the client based on the migration stage. For example:
- Dual-read scenario: The component returns the information of all 16 read partitions from both the source and destination clusters.
- Write-only to destination cluster: The component returns only the eight write partitions of the destination cluster.

Benefits of the migration solution

The ApsaraMQ for RocketMQ cloud migration solution uses a proprietary message routing metadata proxy component developed by Alibaba Cloud. This component supports message routing, scheduling, and traffic shifting control at topic-level granularity, providing the following benefits:

No service interruptions and minimal impact on messaging

The migration solution supports seamless switching. During the traffic switch, messaging is not interrupted, and business applications are not affected. The risk of message latency or duplication is very low.
No additional resources required

Your applications do not need to be scaled out or deployed in multiple clusters. The migration only requires a rolling upgrade of configurations and does not require extra machine resources.
Low business impact and easy implementation

During the migration, you only need to change the endpoint configuration for your applications and perform a single restart. Subsequent traffic switching is automatically handled by dynamic configurations on the ApsaraMQ for RocketMQ server. You can upgrade upstream and downstream applications independently without sorting out messaging dependencies.
Supports canary release and rollback

Migration tasks operate at topic-level granularity. You can perform a canary release for specific topics. If any business risks arise during the migration, you can perform a rollback at any time.