ApsaraMQ for Kafka:Migrate metadata and message data from self-managed Apache Kafka clusters to ApsaraMQ for Kafka instances

Migration process

Usage notes

• If you want to use a non-serverless ApsaraMQ for Kafka instance as the destination instance when you create a migration task, make sure that you select a running instance whose edition is Professional Edition (High Write) or Professional Edition (High Read) and major version is 2.2.0 or later.

• If you want to migrate the metadata and message data of a self-managed Apache Kafka cluster deployed on Alibaba Cloud, we recommend that you purchase an ApsaraMQ for Kafka instance in the same region as the self-managed Apache Kafka cluster, deploy it in the same virtual private cloud (VPC), and then migrate the metadata and message data in the VPC.

• In this topic, the metadata and message data of a self-managed Apache Kafka cluster is migrated to an Internet- and VPC-connected ApsaraMQ for Kafka instance.

  Important

  If you want to migrate a self-managed Apache Kafka cluster to an ApsaraMQ for Kafka instance over the Internet, you must enable Internet access for the instance before you synchronize the data. For information about how to enable Internet access, see (Optional) Enable Internet access.

Step 1: Evaluate specifications

ApsaraMQ for Kafka provides the specification evaluation feature that assesses and recommends the ApsaraMQ for Kafka instance specifications required by a migration task based on the information of the self-managed Apache Kafka cluster, such as cluster traffic, disk capacity, and disk type. For more information, see Evaluate specifications.

Step 2: Purchase an instance

Purchase an ApsaraMQ for Kafka instance based on the evaluated instance specifications and deploy it. For more information, see Purchase and deploy an Internet- and VPC-connected instance.

Step 3: Create a migration task

1. Log on to the ApsaraMQ for Kafka console. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.

2. In the left-side navigation pane, click Migration. On the page that appears, click the Cloud Migration tab.

3. On the Cloud Migration tab, click Create Task.

4. In the Create Cloud Migration Task wizard, configure the parameters.

   1. In the Configure Basic Information step, configure the Task Name and Destination Instance parameters, and click Next.

   2. In the Configure Source Service step, set the Source Instance Type parameter to Public Network (IDC or Cross-cloud Instance) and configure the other parameters. Then, click Next.

      Parameter	Description	Example
      Endpoint	The public endpoint of the self-managed Apache Kafka cluster.	192.168.XX.XX:9092
      Security Protocol	The security protocol of the self-managed Apache Kafka cluster. Valid values: PLAINTEXT SASL_PLAINTEXT SASL Username: Enter the SASL username. SASL Password: Enter the SASL password. Sasl_Mechanism: Select a SASL authentication mechanism. Valid values: PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512. SASL_SSL SASL Username: Enter the SASL username. SASL Password: Enter the SASL password. Sasl_Mechanism: Select a SASL authentication mechanism. Valid values: PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512. SSL Truststore File: Upload the certificate file. SSL Truststore Password: Enter the certificate password. SSL Endpoint Identification Algorithm: Specify the algorithm used to verify server certificates. If you use the SSL protocol for communications, you can configure this parameter to verify server identities to prevent man-in-the-middle attacks. You can enter https, http, or an empty string in this field.	PLAINTEXT
      Number of Tasks	The number of tasks for data synchronization. Valid values: 1 6 12	12
      Synchronize SASL Users	Specifies whether to synchronize the configuration data of the SASL users of the self-managed Apache Kafka cluster to the ApsaraMQ for Kafka instance. This parameter is displayed only after you click Configure Runtime Environment. Default value: No.	Yes
      Synchronize Topic ACLs	Specifies whether to synchronize the configuration data of the access control lists (ACLs) attached to topics in the self-managed Apache Kafka cluster to the ApsaraMQ for Kafka instance. This parameter is displayed only after you click Configure Runtime Environment. Default value: No. Yes: The configuration data of the ACLs attached to topics in the self-managed Apache Kafka cluster is synchronized to the ApsaraMQ for Kafka instance. Before you synchronize the configuration data of the ACLs to the instance, you must create a SASL user on the ApsaraMQ for Kafka instance. No: The configuration data of the ACLs attached to topics in the self-managed Apache Kafka cluster is not synchronized to the ApsaraMQ for Kafka instance.	Yes
      Synchronize Consumer Groups	Specifies whether to synchronize the consumer groups of the self-managed Apache Kafka cluster to the ApsaraMQ for Kafka instance. This parameter is displayed only after you click Configure Runtime Environment. Default value: No.	Yes
      Synchronize Consumer Offsets	Specifies whether to synchronize the consumer offsets of the self-managed Apache Kafka cluster to the ApsaraMQ for Kafka instance. This parameter is displayed only after you click Configure Runtime Environment and set the Synchronize Consumer Groups parameter to Yes. Default value: No.	Yes
      Topic	The topics that you want to synchronize to the ApsaraMQ for Kafka instance. If you do not configure this parameter, all topics in the self-managed Apache Kafka cluster are synchronized. This parameter is displayed only after you click Configure Runtime Environment.	test-topic
      Create Topics to Use Local Storage	The non-log-compacted topics that you want to synchronize to the ApsaraMQ for Kafka instance. If you want the created topics to use local storage, you must configure this parameter. If you do not configure this parameter, the created topics use the cloud storage. This parameter is displayed only after you click Configure Runtime Environment.	test-topic

   3. In the Configure Destination Service step, click Create.

5. On the Cloud Migration tab of the Migration page, select the destination instance from the Instance drop-down list, find the created migration task, and then click Deploy in the Actions column.

   If a migration task is created, you can see that the Status column corresponding to the task becomes Running on the Cloud Migration tab of the Migration page.

Step 4: View progress

1. On the Migration page, click the Cloud Migration tab.

2. Find the task that you want to manage and click Synchronization Progress in the Actions column.

3. In the Select a topic drop-down list in the Synchronization Progress panel, view the synchronized topics.

4. Select the topic that you want to view from the Select a topic drop-down list to check the data synchronization status of each partition of the topic.

What to do next

1. Enable new consumer groups for the ApsaraMQ for Kafka instance to consume messages in the instance.

2. Enable new producers for the ApsaraMQ for Kafka instance, shut down the original producers, and allow the original consumer groups to continue consuming messages in the self-managed Apache Kafka cluster.

3. After all messages in the self-managed Apache Kafka cluster are consumed by the original consumer groups, shut down them and the self-managed Apache Kafka cluster.