ApsaraMQ for Kafka:Migrate the metadata of a self-managed Kafka cluster to a new ApsaraMQ for Kafka instance

Export metadata

Use the metadata export tool to export the metadata of the self-managed Kafka cluster.

1. Click kafka-migration-assessment.jar to download the file that contains the metadata export tool.
2. Upload the file that contains the metadata export tool to your self-managed Kafka cluster.
3. Run the following commands in the directory in which the file is stored to extract the toolkit and allow the JAR file to be executed:

   unzip -o kafka-migration-assessment.jar.zip

   chmod 777 kafka-migration-assessment.jar

4. Run the following command to export the metadata of the self-managed Kafka cluster:

   java -jar kafka-migration-assessment.jar MigrationFromZk --sourceZkConnect <host:port> --sourceBootstrapServers <host:port> --targetDirectory ../xxx/ --fileName metadata.json --commit

   Parameter	Description	Example
   sourceZkConnect	The IP address and port of the ZooKeeper node on which the self-managed Kafka cluster is deployed. If you do not configure this parameter, the tool automatically obtains the IP address and the port number.	192.168.XX.XX:2181
   sourceBootstrapServers	The IP address and port of the self-managed Kafka cluster. If you do not configure this parameter, the tool automatically obtains the IP address and the port number.	192.168.XX.XX:9092
   targetDirectory	The directory in which you want to store the metadata file. If you do not configure this parameter, the metadata file is exported to the current directory by default.	../home/
   fileName	The name of the metadata file. If you do not configure this parameter, the default name kafka-metadata-export.json is used.	metadata.json
   commit	Commits the task.	commit
   installTsar	Specifies whether to install Taobao System Activity Reporter (Tsar). By default, Tsar is not automatically installed. Tsar can be used to obtain accurate information about the specifications of the machine that is used for the self-managed Kafka cluster and the recent memory usage, traffic, and configurations of the cluster. The automatic installation of Tsar requires a long period of time and may fail if the environment is incompatible.	N/A
   evaluate	Specifies whether to obtain the specifications of the machine that is used for the self-managed Kafka cluster and the recent memory usage, traffic, and configurations of the cluster. ApsaraMQ for Kafka can evaluate the specification information and recommend specifications for the destination Message Queue for Apache Kafka instance based on the collected information. We recommend that you create a destination instance based on your business requirements The default value is true. This value specifies that the tool collects the specification information. If you do not want to use the specification evaluation feature, set the value of this parameter to false.	N/A

   After you export the metadata, the system generates a JSON file in the specified directory.

   You can view the metadata file in the specified directory. You also can download and save the file to your on-premises machine.

Step 1: Create a migration task

1. Log on to the ApsaraMQ for Kafka console.
2. In the Resource Distribution section of the Overview page, select the region where your instance is deployed.
3. In the left-side navigation pane, click Migrate.
4. On the Migration to Cloud page, click Create Task.
5. In the Create Task section of the Create Task wizard, configure the parameters and click Next.
   1. Enter a task name in the Description field.
   2. Click Click here to upload a metadata file next to Metadata and upload the metadata file that you exported from the self-manage Apache Kafka cluster. The file must be in JSON format.

6. In the Purchase Instance section of the Create Task wizard, click Buy Instance.

7. In the Select Instance Billing Method panel, set the value of the Billing Method parameter to Subscription or Pay-as-you-go and click OK.

8. On the buy page, configure the Network Type parameter and other parameters based on your business requirements and click Buy Now. Then, follow the on-screen instructions to complete the payment.

   Note

   If you set the value of the evaluate parameter to true when you configure the migration tool to export metadata, the migration tool collects specification information about the self-managed cluster, including the specifications of the server that is used for the cluster and the recent memory usage, traffic, and configurations of the cluster. Message Queue for Apache Kafka evaluates the specification information and recommends specifications for the destination instance based on this information. You can also specify the specifications of the destination instance based on your business requirements.

   On the Instances page, you can view the instance that you purchased.

Step 2: Deploy the instance and start the migration task

1. On the Migration to Cloud page, find the migration task and click Continue with Settings in the Actions column.

2. In the Purchase Instance section, click Next.

3. In the Deploy Instance section, select the destination instance that is created, configure the parameters, and then click Deploy and Create Task.

   Parameter	Description	Example
   Destination Instance	The name of the destination ApsaraMQ for Kafka instance.	alikafka_pre-cn-tl32je2j****
   VPC ID	The ID of the virtual private cloud (VPC) in which you want to deploy the destination instance. To view the ID of the VPC, log on to the VPC console, select the region where the VPC is deployed and click vSwitch in the left-side navigation pane. On the vSwitch page, you can view the VPC ID in the VPC column.	vpc-rj91f8ql28reon76a****
   vSwitch ID	The ID of the vSwitch to which the destination instance is connected. To view the ID of the vSwitch, log on to the VPC console, select the region where the VPC is deployed and click vSwitch in the left-side navigation pane. On the vSwitch page, you can view the ID of the vSwitch in the Instance ID/Name column. After you select a vSwitch ID, the system automatically selects the zone where the vSwitch is located.	vsw-rj9dfeeir1xl8px4g****
   Cross-zone Deployment	Specify whether to deploy the destination instance across multiple zones. This parameter is available for Professional Edition instances. Yes: The system deploys the destination instance in multiple zones. Instances that are deployed across multiple zones provide high disaster recovery capabilities and have high tolerance for faults that occur in data centers. No: The system deploys the destination instance only in the current zone.	N/A
   Version	The broker version of the ApsaraMQ for Kafka instance that you want to deploy. The version number must be the same as the version number of the broker that the self-managed Kafka cluster uses.	2.2.0
   Message Retention Period	The maximum retention period of messages when disk capacity is sufficient. When the disk usage reaches 85%, the system deletes messages in the order in which they are stored, starting from the earliest stored message. This ensures that the performance of the service is not degraded.	72
   Maximum Message Size	The maximum size of a message that can be sent and received by the ApsaraMQ for Kafka instance. Make sure that the value of this parameter is the same as the maximum message size that is specified for the producers and consumers.	1
   Consumer Offset Retention Period	The retention period of consumer offsets. Default value: 10080. Unit: minutes. Valid values: 1440 to 43200. The default value equals seven days.	10080
   Custom Username and Password	Specify whether to use a custom username and password for the destination instance. If the destination instance can be accessed over the Internet and a VPC, this parameter is available. Yes: You specify a username and password for the instance. This method is suitable for scenarios in which multiple instances use the same username and password. No: The system generates a username and password for the instance.	No
   ACL	Specify whether to enable the access control list (ACL) feature for the instance. Enable: The ACL feature is enabled. The ACL feature of ApsaraMQ for Kafka allows you to authorize Simple Authentication and Security Layer (SASL) users to publish messages to the ApsaraMQ for Kafka instance and consume messages in the instance based on your business requirements. For information about how to grant permissions to SASL users, see Grant permissions to SASL users. Disable: The ACL feature is disabled. All SASL users of the instance can publish messages to the Message Queue for Apache Kafka instance and consume messages in the instance. Message Queue for Apache Kafka only authenticates the identifies of the SASL users.	N/A

   On the Migrate page, you can view that the task is in the Migrating state. After the migration task is complete, you can view the topics and groups that are created in the destination instance.

Delete a migration task

On the Migrate page, find the migration task that you want to delete and click Delete in the Actions column.

What to do next

View the migration progress