Import metadata - ApsaraMQ for Kafka - Alibaba Cloud Documentation Center

This topic describes how to create a migration task in the ApsaraMQ for Kafka console to migrate the metadata of a self-managed Apache Kafka cluster to an ApsaraMQ for Kafka instance.

Migration process

Usage notes

You cannot create metadata import tasks for serverless ApsaraMQ for Kafka instances.
If the self-managed Apache Kafka cluster whose metadata you want to migrate does not support Internet access, we recommend that you use the metadata import feature to migrate the metadata. Otherwise, we recommend that you use the cloud migration feature to migrate the metadata. For more information, see Migrate metadata and message data from self-managed Apache clusters to ApsaraMQ for Kafka instances.
The metadata import feature imports only topic and group configurations from a self-managed Apache Kafka cluster to an ApsaraMQ for Kafka instance. This feature is suitable for scenarios where message data is not migrated. If you want to migrate both message data and metadata, use the cloud migration feature. For more information, see Migrate metadata and message data from self-managed Apache clusters to ApsaraMQ for Kafka instances.
In this topic, the metadata of a self-managed Apache Kafka cluster that does not support Internet access is migrated.

Step 1: Evaluate specifications

ApsaraMQ for Kafka provides the specification evaluation feature that assesses and recommends the ApsaraMQ for Kafka instance specifications required by a migration task based on the information of the self-managed Apache Kafka cluster, such as cluster traffic, disk capacity, and disk type. For more information, see Evaluate specifications.

Step 2: Purchase an instance

Purchase an ApsaraMQ for Kafka instance based on the evaluated specifications and deploy it. When you purchase the instance, determine whether to enable Internet access based on your business requirements.

For information about how to purchase and deploy an Internet- and virtual private cloud (VPC)-connected instance, see Purchase and deploy an Internet- and VPC-connected instance.
For information about how to purchase and deploy a VPC-connected instance, see Purchase and deploy a VPC-connected instance.

Step 3: Export metadata

Log on to the server of the self-managed Apache Kafka cluster, download and install JDK 8 or 11. For more information, see Java Downloads.
Download the export tool kafka-migration-assessment.jar.
Run the following command to grant the executable permission on the JAR file:
```
chmod 777 kafka-migration-assessment.jar
```

Run the following command to export the metadata of the self-managed Apache Kafka cluster:

java -jar kafka-migration-assessment.jar MigrationFromZk --sourceZkConnect <yourHost:yourPort> --sourceBootstrapServers <yourHost:yourPort> --targetDirectory <yourDirectoryName> --fileName <yourFileName> --commit

Parameter	Description	Example
sourceZkConnect	The IP address and port of the ZooKeeper node on which the self-managed Apache Kafka cluster is deployed. If you do not configure this parameter, the metadata export tool automatically obtains the IP address and the port number.	192.168.XX.XX:2181
sourceBootstrapServers	The IP address and port of the self-managed Apache Kafka cluster. If you do not configure this parameter, the metadata export tool automatically obtains the IP address and the port number.	192.168.XX.XX:9092
targetDirectory	The directory in which you want to store the metadata file. If you do not configure this parameter, the metadata file is exported to the current directory by default.	../home/
fileName	The name of the metadata file. If you do not configure this parameter, the default name kafka-metadata-export.json is used.	metadata.json
commit	Commits the task.	commit
installTsar	Specifies whether to install Taobao System Activity Reporter (Tsar). By default, Tsar is not automatically installed. Tsar can be used to obtain accurate information about the specifications of the machine that is used to run the self-managed Apache Kafka cluster and the recent memory usage, traffic usage, and configurations of the cluster. The automatic installation of Tsar requires a long period of time to complete and may fail if the environment is incompatible.	None
evaluate	Specifies whether to obtain the information about the specifications of the machine that is used to run the self-managed Apache Kafka cluster and the recent memory usage, traffic, and configurations of the cluster for specification evaluation. Valid values: true and false. Default value: true. If you set this parameter to true, the ApsaraMQ for Kafka console evaluates and recommends the specifications of the ApsaraMQ for Kafka instance to which the metadata is migrated. If you do not want to use the specification evaluation feature, set this parameter to false.	None

After you export the metadata, the system generates a JSON file in the specified directory.

View the metadata file in the specified directory. You also can download and save the file to your on-premises machine.

Step 4: Create a metadata import task

Log on to the ApsaraMQ for Kafka console. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.
In the left-side navigation pane, click Migration. On the page that appears, click the Metadata Import tab.
On the Metadata Import tab, click Create Task.
In the Create Metadata Import Task panel, configure the parameters and click Create.
1. In the Create Task step, configure the Task Name and Destination Instance parameters and upload the metadata file that is obtained in Step 2: Export the metadata.
2. In the Edit Topic step, perform the following operations:
  - Add a topic: Click Add Topic. In the panel that appears, configure the Name, Description, Partitions, and Message Type parameters.
  - Modify a topic: Find the topic that you want to modify and click Modify in the Actions column. In the panel that appears, configure the Description, Partitions, Message Type, and Log Cleanup Policy parameters.
  - Delete a topic: Find the topic that you want to delete and click Delete in the Actions column. In the Note message, click OK.
3. In the Edit Group step, perform the following operations:
  - Add a group: Click Add Group. In the panel that appears, configure the Group ID and Description parameters.
  - Modify a group: Find the group that you want to modify and click Modify in the Actions column. In the panel that appears, configure the Description parameter.
  - Delete a group: Find the group that you want to delete and click Delete in the Actions column. In the Note message, click OK.
After the metadata import task is created, you can view the migration progress of the topics and groups in the Topic Migration Progress and Group Migration Progress columns on the Migration page. You can also click Details in the Actions column to view the details of the metadata import task.

Step 5: View the migration result

Log on to the ApsaraMQ for Kafka console. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.
In the left-side navigation pane, click Instances.
On the Instances page, click the name of the instance that you want to manage.
View the topics and groups on the instance.
- In the left-side navigation pane, click Topics. On the page that appears, view the created topics on the instance.
- In the left-side navigation pane, click Groups. On the page that appears, view the created groups on the instance.

What to do next

Enable new consumer groups for the ApsaraMQ for Kafka instance to consume messages in the instance.
Enable new producers for the ApsaraMQ for Kafka instance, shut down the original producers, and allow the original consumer groups to continue consuming messages in the self-managed Apache Kafka cluster.
After all messages in the self-managed Apache Kafka cluster are consumed by the original consumer groups, shut down them and the self-managed Apache Kafka cluster.