Global Replicator - ApsaraMQ for RocketMQ - Alibaba Cloud Documentation Center

This topic describes the operation guide, page parameters, and frequently asked questions (FAQ) for Global Message Replication, so that you can quickly understand and use this feature.

Prerequisites

An ApsaraMQ for RocketMQ (cloud message queue) instance is created. Version requirements:

ApsaraMQ for RocketMQ 4.0 series instances.
Note
The Standard Edition of 4.0 series instances supports only one-way synchronization.
ApsaraMQ for RocketMQ 5.0 series instances.

This topic uses ApsaraMQ for RocketMQ 5.0 series instances as both the source cluster and the target cluster as an example.

Specification recommendations

When you use Global Message Replication, message synchronization adds TPS overhead to the instances after you configure topic mappings. You must therefore account for the peak sending and receiving TPS limits of the source and target ApsaraMQ for RocketMQ instances of the synchronization task.

Recommended specifications:

For example, without synchronization, an ApsaraMQ for RocketMQ instance has a peak sending TPS of 10,000 messages/s and a peak receiving TPS of 10,000 messages/s.

The following TPS configurations are recommended for the source instance and the target instance:

Task type	Source instance			Target instance
Task type	Peak sending and receiving TPS limit (messages/s)	Peak sending TPS (messages/s)	Peak receiving TPS (messages/s)	Peak sending and receiving TPS limit (messages/s)	Peak sending TPS (messages/s)	Peak receiving TPS (messages/s)
One-way synchronization	30,000	10,000	20,000	20,000	10,000	10,000
Two-way synchronization	40,000	20,000	20,000	40,000	20,000	20,000

Note

You can view the peak sending and receiving TPS limit of an instance on the Instance Details page. For the TPS limits of instances of different specifications, see Specification limits.
You can adjust the peak receiving TPS of the target instance based on the processing speed that your business requires.

Create a Global Message Replication task

Note

When you use Global Message Replication for the first time, the Alibaba Cloud Service Authorization - RocketMQ Global Message Replication dialog box appears. After you click Confirm, the service-linked role AliyunServiceRoleForRMQDisasterRecovery of the target service is automatically created, so that the Global Message Replication feature can run in ApsaraMQ for RocketMQ. For more information, see service-linked roles.

Step 1: Create a task

Log on to the ApsaraMQ for RocketMQ console.
In the left-side navigation pane, click Global Message Replication and then click Create Task.

On the Create Task page, enter the following information. The main configurations are described below. Configure the remaining parameters as needed. Then click Create.

At the top of the page, set Task Name to rocket-test, select One-way Backup: synchronizes data from the source cluster to the destination cluster for Task Type, and enter Task Description as needed. Configure the source cluster and the target cluster in the left and right panes. An arrow between the panes indicates the synchronization direction.

For the configurations of the different cluster types of the source cluster and the target cluster, see the following tables:

ApsaraMQ for RocketMQ

Parameter	Description	Example
Version	Select the version of the ApsaraMQ for RocketMQ instance. 4.0 series instance 5.0 series instance	5.0 series instance
Region	Select the region of the ApsaraMQ for RocketMQ instance.	China (Hangzhou)
Instance	Select the ApsaraMQ for RocketMQ instance.	rmq-cn-778***
Authentication mode	No Configuration Required ACL-based authentication	ACL-based authentication
Username	Select a username.	yS8x01****
Security group ID	The security group of the ECS instance that hosts the ApsaraMQ for RocketMQ instance. This parameter is required when Version is set to a 5.0 series instance. Inbound rule: No requirements. Outbound rule: Policy: Allow Protocol: Custom TCP Destination: The CIDR block must include all CIDR blocks of the cluster's vSwitches. The port range must include 8080 and 8081.	sg-2ze4jlbqy2s40pc4****
Filter condition system configuration	(Optional) You can add custom attributes (user properties) to messages. Consumers can then filter messages based on a subscription policy that uses SQL-92 syntax. Key: The key of the user property. Source cluster value: The value of the user property that is added to messages from the source cluster. Target cluster value: The value of the user property that is added to messages from the target cluster. The following code provides an example of SQL-92 subscription filtering: `String sqlStr = "{Key} IS NOT NULL AND {Key} = {Source/Target Cluster Value}"; consumer.subscribe(MqConfig.TOPIC, MessageSelector.bySql(sqlStr));`	None

Apache RocketMQ

Parameter	Description	Example
Access point	The format for an access point is `{domain name/IP}:{port number}`.	XX.XX.XX.XX:8080
Authentication mode	No Configuration Required ACL-based authentication	ACL-based authentication
Username	Enter a username based on your business requirements.	yS8x01****
Password	Enter a password based on your business requirements.	F17R4to****
Network type	Public network VPC	Public network
Region	This parameter is required if you set Network type to VPC. Select the region of the ECS instances on which Apache RocketMQ is deployed.	China (Hangzhou)
VPC	This parameter is required if you set Network type to VPC. Select the VPC of the ECS instances on which Apache RocketMQ is deployed.	vpc-bp17fapfdj0dwzjkd****
vSwitch	This parameter is required if you set Network type to VPC. Select the vSwitch of the ECS instances on which Apache RocketMQ is deployed.	vsw-bp1gbjhj53hdjdkg****
Security group ID	This parameter is required if you set Network type to VPC. Select the security group of the ECS instances on which Apache RocketMQ is deployed. Inbound rule: No requirements. Outbound rule: Policy: Allow Protocol: Custom TCP Destination: The CIDR block must include the CIDR blocks of all Apache RocketMQ nodes. The port range must include the service ports of all Apache RocketMQ nodes. The default ports are 9876, 10911, and 10909. If you have modified the NameServer or broker ports, you must specify the updated ports.	sg-2ze4jlbqy2s40pc4****
Filter condition system configuration	(Optional) You can add custom attributes (user properties) to messages. Consumers can then filter messages based on a subscription policy that uses SQL-92 syntax. Key: The key of the user property. Source cluster value: The value of the user property that is added to messages from the source cluster. Target cluster value: The value of the user property that is added to messages from the target cluster. The following code provides an example of SQL-92 subscription filtering: `String sqlStr = "{Key} IS NOT NULL AND {Key} = {Source/Target Cluster Value}"; consumer.subscribe(MqConfig.TOPIC, MessageSelector.bySql(sqlStr));`	None

Step 2: Configure topic mappings

On the Task Basic Information page, click Message Synchronization Tasks in the left-side navigation pane.
Click Batch Mapping or Add Mapping to configure topic mappings for synchronization in the dialog box.
- Batch Mapping: Configure and select the required topic mappings, and then click OK.
- Add Mapping: Configure the required topic mapping, and then click OK.
Note
- If the source cluster is an Apache RocketMQ cluster, you must specify a group for message synchronization. Do not use a group that is already in use for business operations.
- Message synchronization starts only after you configure the topic mappings.

Step 3: (Optional) Enable consumer progress synchronization

After you enable consumer progress synchronization, the current consumption progress of a group that consumes a topic can be synchronized to the target cluster. For more information, see Consumer progress synchronization.

Consumer client configuration

For example, you configure a two-way synchronization task in which the source cluster is an ApsaraMQ for RocketMQ instance in the China (Shanghai) region and the target cluster is an ApsaraMQ for RocketMQ instance in the China (Hangzhou) region. The Key of the filter condition is set to Region, the source cluster value is set to Shanghai, and the target cluster value is set to Hangzhou.

Under normal conditions, a consumer client in Shanghai consumes only the data of the Shanghai region and does not consume the data of Hangzhou. Matching by a single custom attribute is therefore sufficient. For more information, see Message filtering.

Subscribe to messages and match them based on a single custom attribute.

String topic = "topic";
// Subscribe only to the messages of the local region.
FilterExpression filterExpression = new FilterExpression("Region IS NOT NULL", FilterExpressionType.SQL92);
// The Region key in the user property is NULL for locally produced messages because the local producer application does not set this key.
simpleConsumer.subscribe(topic, filterExpression);

Subscribe to all messages in the topic without filtering.

String topic = "topic";
// Subscribe to all messages.
FilterExpression filterExpression = new FilterExpression("True", FilterExpressionType.SQL92);
simpleConsumer.subscribe(topic, filterExpression);

FAQ

Can I synchronize data among three instances?

Yes. A Global Message Replication task is configured between two instances. To synchronize data among three instances, you must configure three tasks. This ensures that each instance holds the complete dataset.

Can I use tag-based filtering for instances in a two-way synchronization setup?

You can use either tag-based filtering or SQL filtering, but not both. Because consumer applications must use SQL filtering to filter data, we recommend that instances that would use tag-based filtering use SQL filtering for message filtering instead. For more information, see Message filtering.

Can I change an existing one-way synchronization task to a two-way task?

If both the source cluster and the target cluster are ApsaraMQ for RocketMQ instances, the change is supported and does not affect the running of existing one-way tasks. If one of the clusters is an open source Apache RocketMQ cluster, changing a one-way synchronization task to a two-way synchronization task is not supported. You may need to delete the task and create it again.

How is Global Message Replication billed?

The synchronization channel of Global Message Replication depends on EventBridge. For the pricing of EventBridge, see EventBridge. When Global Message Replication synchronizes message data, it consumes the read performance of the source ApsaraMQ for RocketMQ instance and the write performance of the target instance. You must additionally evaluate the computing specifications of ApsaraMQ for RocketMQ.

I configured ACL-based authentication for Global Message Replication. Why are synchronized messages not received?

After you configure a synchronization task in Global Message Replication, you must specify the topics to synchronize. The system creates a non-deletable system group for each topic to synchronize data. This group is deleted only when the synchronization task is deleted. If you enable ACL-based authentication, you must grant the selected ACL user read and write permissions on the topics and read permissions on the groups. On the Access Control page of the instance, select the user and set Resource name to a wildcard character (*) to grant full permissions to the account. This simplifies the configuration.

If I pause and then resume a task, will synchronization continue from where it left off?

Yes. After resumption, synchronization continues from the message following the latest acknowledged offset. Ensure the pause duration does not exceed the message retention period of the ApsaraMQ for RocketMQ instance.