ApsaraMQ for Kafka provides a fully managed and O&M-free cloud migration feature that allows you to synchronize data on a self-managed or cross-cloud instance to an ApsaraMQ for Kafka instance. This topic describes the source instance types and limits of the cloud migration feature. This topic also describes how to perform data synchronization across virtual private clouds (VPCs).
Prerequisites
An ApsaraMQ for Kafka instance that meets the conditions that are described in the following table is purchased and deployed.
Item | Description |
Status | Running |
Version | The version of the ApsaraMQ for Kafka instance must meet one of the following requirements:
|
You can view the status and version of an instance in the Basic Information section of the Instances page in the ApsaraMQ for Kafka console.
Background information
The cloud migration feature of ApsaraMQ for Kafka allows you to synchronize the metadata and message data of a source cluster to a destination cluster. The metadata includes the configurations of topics and groups. After a cloud migration task is complete, the metadata in the destination cluster is consistent with the metadata in the source cluster and continuously updated.
Limits
The limits that are described in the following table are imposed on the cloud migration feature of ApsaraMQ for Kafka.
Item | Limit |
Quantity | Up to 20 cloud migration tasks can be created on an ApsaraMQ for Kafka instance. The total number of cloud migration tasks and connector tasks cannot exceed 20 on the ApsaraMQ for Kafka instance. |
Instance edition | You can create cloud migration tasks for instances whose edition is Professional Edition (High Write) or Professional Edition (High Read). |
Billing
The cloud migration feature of ApsaraMQ for Kafka is in public preview and independent of ApsaraMQ for Kafka instances. ApsaraMQ for Kafka does not charge you for the feature. Alibaba Cloud does not provide a service level agreement (SLA) for the cloud migration feature. For information about the SLAs and billing of other Alibaba Cloud services that are required to use the cloud migration feature, see the corresponding documentation.
Environment requirements
If you want to use the cloud migration feature to synchronize data on a self-managed instance that is connected over the Internet to an ApsaraMQ for Kafka instance or synchronize data on an ApsaraMQ for Kafka instance that is deployed in a VPC to an ApsaraMQ for Kafka instance that is deployed in another VPC, you must enable Internet access for the destination instance and synchronize data over the Internet. For more information, see (Optional) Enable Internet access.
If you use the cloud migration feature to synchronize data on an ApsaraMQ for Kafka instance that is deployed in a VPC to an ApsaraMQ for Kafka instance that is deployed in another VPC, you must connect the VPCs. Automatic VPC connection is supported only in the China (Hangzhou) and China (Chengdu) regions. For information about how to connect VPCs in regions other than the China (Hangzhou) and China (Chengdu) regions, see Use Basic Edition transit routers to connect VPCs across regions.
(Optional) Enable Internet access
If you want to use the cloud migration feature to access other Alibaba Cloud services or Internet endpoints across regions, you must enable Internet access.
Create a NAT gateway for VPC 1 in which the ApsaraMQ for Kafka instance is deployed.
For more information, see Create and manage Internet NAT gateways.
Bind an elastic IP address (EIP) to the created NAT gateway.
For more information, see Associate an EIP with an ECS instance.
Create an SNAT entry for the vSwitch that is used by the ApsaraMQ for Kafka instance that is deployed in VPC 1.
For more information, see Create and manage SNAT entries.
Create and deploy a cloud migration task
Log on to the ApsaraMQ for Kafka console. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.
In the left-side navigation pane, click Migration. On the page that appears, click the Cloud Migration tab.
On the Cloud Migration tab, click Create Task.
In the Create Cloud Migration Task panel, configure the parameters.
In the Configure Basic Information step, configure the Task Name and Destination Instance parameters and click Next.
In the Configure Source Service step, configure the parameters and click Next.
If you set the Source Instance Type parameter to VPC (Interconnection Between Self-managed Instance and VPC), configure the parameters that are described in the following table.
Parameter
Description
Example
Source Region
The ID of the region in which the source instance is deployed.
China (Hangzhou)
Endpoint
The endpoint of the source instance.
192.168.XX.XX:9092
VPC ID
The ID of the VPC in which the source instance is deployed. By default, the VPC ID that you specified when you deployed the source instance is displayed. You do not need to configure this parameter.
vpc-test-c****
vSwitch ID
The ID of the vSwitch that is used by the source instance.
vsw-bp1gbjqsf53og****
Security Group
The security group to which the source instance belongs.
alikafka_pre-cn-7mz2****
Security Protocol
The security protocol that is used by the source instance.
PLAINTEXT
SASL_PLAINTEXT
SASL Username: Enter the Simple Authentication and Security Layer (SASL) username.
SASL Password: Enter the SASL password.
Sasl_Mechanism: The SASL authentication mechanism. Valid values: PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512.
SASL_SSL
SASL Username: Enter the SASL username.
SASL Password: Enter the SASL password.
Sasl_Mechanism: The SASL authentication mechanism. Valid values: PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512.
SSL Truststore File: Upload the Security Socket Layer (SSL) certificate file.
SSL Truststore Password: Enter the password of the SSL certificate.
SSL Endpoint Identification Algorithm: The SSL attribute that is used to specify the algorithm for verifying the server certificate. If you use the SSL protocol for communications, you can configure this parameter to verify the server identity to prevent man-in-the-middle (MITM) attacks. You can enter https, http, or an empty string in this field.
PLAINTEXT
Number of Tasks
The number of threads that you want to use to synchronize data. Valid values:
1
6
12
12
Synchronize SASL Users
Specifies whether to synchronize the configuration data of the SASL users of the source instance to the destination instance. This parameter is displayed only if you click Configure Runtime Environment. Default value: Yes.
Yes
Synchronize Topic ACLs
Specifies whether to synchronize the configuration data of the access control lists (ACLs) that are attached to the topics on the source instance to the destination instance. This parameter is displayed only if you click Configure Runtime Environment. Default value: Yes. Valid values:
Yes: The configuration data of the ACLs that are attached to the topics on the source instance is synchronized to the destination instance. If the source instance is a self-managed instance, you must create an SASL user on the destination instance before you synchronize the configuration data of the ACLs that are attached to the topics on the source instance to the destination instance.
No: The configuration data of the ACLs that are attached to the topics on the source instance is not synchronized to the destination instance.
Yes
Synchronize Consumer Groups
Specifies whether to synchronize the consumer groups on the source instance to the destination instance. This parameter is displayed only if you click Configure Runtime Environment. Default value: Yes.
Yes
Synchronize Consumer Offsets
Specifies whether to synchronize the consumer offsets of the source instance to the destination instance. This parameter is displayed only if you click Configure Runtime Environment and set the Synchronize Consumer Groups parameter to Yes. Default value: Yes.
Yes
Topic
The topics that you want to synchronize to the destination instance. If you do not configure this parameter, the configuration data of all topics on the source instance is synchronized to the destination instance. This parameter is displayed only if you click Configure Runtime Environment.
test-topic
Create Topics to Use Local Storage
The non-compacted topics that you want to migrate to the destination instance. If you want to use the local storage of the destination instance to store data in the topics, you must configure this parameter. If you do not configure this parameter, the system stores data in all topics by using the cloud storage. This parameter is displayed only if you click Configure Runtime Environment.
test-topic
If you set the Source Instance Type parameter to Public Network (IDC or Cross-cloud Instance), configure the parameters that are described in the following table.
Parameter
Description
Example
Endpoint
The endpoint of the source instance.
192.168.XX.XX:9092
Security Group
The security group to which the source instance belongs.
alikafka_pre-cn-7mz2****
Security Protocol
The security protocol that is used by the source instance.
PLAINTEXT
SASL_PLAINTEXT
SASL Username: Enter the SASL username.
SASL Password: Enter the SASL password.
Sasl_Mechanism: The SASL authentication mechanism. Valid values: PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512.
SASL_SSL
SASL Username: Enter the SASL username.
SASL Password: Enter the SASL password.
Sasl_Mechanism: The SASL authentication mechanism. Valid values: PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512.
SSL Truststore File: Upload the SSL certificate file.
SSL Truststore Password: Enter the password of the SSL certificate.
SSL Endpoint Identification Algorithm: The SSL attribute that is used to specify the algorithm for verifying the server certificate. If you use the SSL protocol for communications, you can configure this parameter to verify the server identity to prevent MITM attacks. You can enter https, http, or an empty string in this field.
PLAINTEXT
Number of Tasks
The number of threads that you want to use to synchronize data. Valid values:
1
6
12
12
Synchronize SASL Users
Specifies whether to synchronize the configuration data of the SASL users of the source instance to the destination instance. This parameter is displayed only if you click Configure Runtime Environment. Default value: Yes.
Yes
Synchronize Topic ACLs
Specifies whether to synchronize the configuration data of the ACLs that are attached to the topics on the source instance to the destination instance. This parameter is displayed only if you click Configure Runtime Environment. Default value: Yes.
Yes: The configuration data of the ACLs that are attached to the topics on the source instance is synchronized to the destination instance. If the source instance is a self-managed instance, you must create an SASL user on the destination instance before you synchronize the configuration data of the ACLs that are attached to the topics on the source instance to the destination instance.
No: The configuration data of the ACLs that are attached to the topics on the source instance is not synchronized to the destination instance.
Yes
Synchronize Consumer Groups
Specifies whether to synchronize the consumer groups on the source instance to the destination instance. This parameter is displayed only if you click Configure Runtime Environment. Default value: Yes.
Yes
Synchronize Consumer Offsets
Specifies whether to synchronize the consumer offsets of the source instance to the destination instance. This parameter is displayed only if you click Configure Runtime Environment and set the Synchronize Consumer Groups parameter to Yes. Default value: Yes.
Yes
Topic
The topics that you want to synchronize to the destination instance. If you do not configure this parameter, the configuration data of all topics on the source instance is synchronized to the destination instance. This parameter is displayed only if you click Configure Runtime Environment.
test-topic
Create Topics to Use Local Storage
The non-compacted topics that you want to migrate to the destination instance. If you want to use the local storage of the destination instance to store data in the topics, you must configure this parameter. If you do not configure this parameter, the system stores data in all topics by using the cloud storage. This parameter is displayed only if you click Configure Runtime Environment.
test-topic
In the Configure Destination Service step, click Create.
After the cloud migration task is created, go to the Migration page and select the destination ApsaraMQ for Kafka instance from the Instance drop-down list. Then, find the cloud migration task that you created and click Deploy in the Actions column.
If the value in the Status column on the Migration page changes to Running, the cloud migration task is created.
Other operations
On the Migration page, select the destination ApsaraMQ for Kafka instance from the Instance drop-down list, find the migration task that you want to manage, and perform other operations in the Actions column.
View task details: Click Details. On the Task Details page that appears, you can view the basic information, source service, destination service, and runtime environment of the task.
View the synchronization progress: Click Synchronization Progress, select the topic whose synchronization progress you want to view, and then click OK.
Change task configurations: Click Modify Configuration. In the panel that appears, modify the values of the parameters based on your business requirements.
Reset the consumer offset: Choose . In the panel that appears, select the topic for which you want to reset the consumer offset, specify a new consumer offset, and then click OK. For more information, see Reset consumer offsets.
Stop and start the task: Choose or . In the Note message that appears, click OK.
Delete the task: Choose . In the Note message that appears, click OK.