ApsaraMQ for Kafka:Cloud migration

Migration process

Usage notes

• When you create a Cloud Migration task, a non-Serverless destination instance must be a Professional Edition (High Write) or Professional Edition (High Read) instance. Additionally, its major version must be greater than 2.2.0, and its status must be Running.

• If your self-managed Kafka cluster is also deployed on Alibaba Cloud, we recommend that you purchase a Message Queue for Apache Kafka instance in the same region and VPC to migrate over the internal network.

• This topic uses a public network migration to a Message Queue for Apache Kafka Internet and VPC Type as an example.

  Important

  To migrate data over the public network, you must enable Internet access for the destination instance before you start data synchronization. For more information, see Enable Internet access (optional).

Step 1: Evaluate specifications

The specification evaluation feature assesses your self-managed Kafka cluster's traffic, disk capacity, and disk type and recommends appropriate Message Queue for Apache Kafka instance specifications. For more information, see Run the specification evaluation.

Step 2: Purchase an instance

Based on the recommended specifications, purchase and deploy a Message Queue for Apache Kafka instance. For more information, see Deploy an internet- and VPC-connected instance.

Step 3: Create a Cloud Migration task

1. Log on to the ApsaraMQ for Kafka console. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.

2. In the left-side navigation pane, click Migration. On the page that appears, click the Cloud Migration tab.

3. On the Cloud Migration tab, click Create Task.

4. In the Create Cloud Migration Task panel, configure the following parameters.

   1. On the Configure Basic Information tab, set the Task Name and Destination Instance, and then click Next.

   2. On the Configure Source Service tab, set Source Instance Type to Public Network (IDC or Cross-cloud Instance) and configure the following parameters. Then, click Next.

      Parameter	Description	Example
      Endpoint	The public network endpoint of the self-managed Kafka cluster.	192.168.XX.XX:9092
      Security Protocol	The security protocol of your self-managed Kafka cluster. PLAINTEXT SASL_PLAINTEXT SASL Username: Enter the SASL username. SASL Password: Enter the SASL password. Sasl_Mechanism: The SASL authentication mechanism. Valid values: PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512. SASL_SSL SASL Username: Enter the SASL username. SASL Password: Enter the SASL password. Sasl_Mechanism: The SASL authentication mechanism. Valid values: PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512. SSL Truststore File: Upload the certificate file. SSL Truststore Password: Enter the certificate password. SSL Endpoint Identification Algorithm: The algorithm for server certificate verification, used to prevent man-in-the-middle attacks. Valid values: https, http, or an empty string.	PLAINTEXT
      Number of Tasks	The number of tasks for data synchronization. Valid values: 1 6 12	12
      Synchronize SASL Users	Whether to synchronize SASL user information from the source instance to the Message Queue for Apache Kafka instance during migration. This parameter is displayed after you click Configure Runtime Environment. Default value: No.	Yes
      Synchronize Topic ACLs	Whether to synchronize topic ACLs during migration. Click Configure Runtime Environment to display this parameter. Default value: No. Yes: Synchronizes topic ACLs from the self-managed Kafka cluster to the Message Queue for Apache Kafka instance. You must manually create the corresponding SASL users in the destination Message Queue for Apache Kafka instance before synchronization. No: Topic ACLs from the self-managed Kafka cluster are not synchronized.	Yes
      Synchronize Consumer Groups	Whether to create the consumer groups of the self-managed Kafka cluster on the destination instance during migration. Click Configure Runtime Environment to display this parameter. Default value: No.	Yes
      Synchronize Consumer Offsets	Whether to create the consumer offsets from your self-managed Kafka cluster in the destination instance during migration. This parameter is displayed if you click Configure Runtime Environment and set Synchronize Consumer Groups to Yes. Default value: No.	Yes
      Topic	The topics to synchronize from the self-managed Kafka cluster to the destination instance. If left blank, all topics are synchronized. Click Configure Runtime Environment to display this parameter.	test-topic
      Create Topics to Use Local Storage	The non-Compact topics to retain with local storage on the destination instance. If left empty, these topics are created with cloud storage. Click Configure Runtime Environment to display this parameter.	test-topic

   3. On the Configure Destination Service tab, click Create.

5. After the task is created, on the Migration page, select the destination instance from the Instance drop-down list. Find the task you created in the list and click Deploy in the Actions column.

   On the Migration page, you can see that the Status of the task is Running, indicating that the task is created successfully.

Step 4: View progress

1. On the Migration page, click the Cloud Migration tab.

2. Find the target task and click Synchronization Progress in the Actions column.

3. In the Synchronization Progress panel, the Select a topic drop-down list shows the synchronized topics.

4. Select a topic to view the data synchronization status for each of its partitions.

Next steps

1. Create new consumer groups for the Message Queue for Apache Kafka instance to start consuming messages from it.

2. Create and switch producers to the Message Queue for Apache Kafka instance, decommission the old producers, and let the old consumer groups finish consuming any remaining messages from the self-managed Kafka cluster.

3. After all messages in the self-managed Kafka cluster are consumed, decommission the old consumer groups and the cluster.