ApsaraMQ for Kafka:Metadata import

Prerequisites

• Java Development Kit (JDK) 8 is downloaded and installed.

• Purchase and deploy an ApsaraMQ for Kafka instance:

  • Purchase and deploy a VPC instance

  • Purchase and deploy an internet- and VPC-connected instance

  Note

  For information about the instance specifications to purchase, see Step 1: Evaluate specifications.

Limitations

Serverless instances do not support metadata import tasks.

Background information

Kafka cluster metadata consists of topic and group configurations. This metadata is stored in ZooKeeper, and each node retrieves the latest version from ZooKeeper, so metadata exported from any node is always up-to-date and consistent. You can export the metadata to a JSON file and import it into another Kafka cluster to back up your self-managed cluster.

To migrate metadata, export it from your self-managed cluster and import it into an ApsaraMQ for Kafka instance. ApsaraMQ for Kafka parses the imported metadata so you can visually configure topics and groups. When the import task completes, the corresponding topics and groups are created in the destination ApsaraMQ for Kafka instance.

Step 1: Evaluate specifications

Get instance specification recommendations based on your self-managed Kafka cluster's data.

1. Log on to the ApsaraMQ for Kafka console. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.

2. In the left-side navigation pane, click Migration. On the page that appears, click the Metadata Import tab.

3. On the Metadata Import tab, click Evaluate Specifications.

4. In the Evaluate Specifications panel, enter your self-managed cluster information to get instance specification recommendations for ApsaraMQ for Kafka.

   Parameter	Description	Example
   Broker Version	The Kafka version of your self-managed cluster.	0.10.x
   Migration from IDC	Whether the migration is from an on-premises data center (IDC).	Yes
   Maximum Cluster Traffic	The peak traffic for message production or consumption. Unit: MB/s.	20
   Public Traffic	If you do not need internet access, enter 0. Unit: MB/s.	0
   SSD	Whether the disk type is SSD.	Yes
   Disk Capacity	The total storage capacity of the disk. Unit: GB.	500
   Partitions	The total number of partitions for the topics to be migrated.	800
   Replicas	The number of replicas for each topic.	3
   Scenario	The scenario for message production and consumption.	Business Message
   Response Latency	The response time (RT) requirement for message production and consumption.	No Special Requirements

   After you enter the parameters, click OK. The recommended instance specifications are displayed at the top of the Evaluate Specifications panel. An example of the recommended specifications is as follows: Specification Type is Standard (High-write), Instance Type is VPC instance, Peak Traffic is 20 MB/s, Disk Type is SSD, Disk Capacity is 900 GB, Partitions is 0, and Server Version is 0.10.2. In the instance list, click Purchase Instance and purchase an ApsaraMQ for Kafka instance based on the recommendations.

Step 2: Export metadata

Export the metadata file from your self-managed Kafka cluster.

1. Download the metadata export tool: kafka-migration-assessment.jar.

2. Upload the tool to a server in your self-managed Kafka cluster.

3. Run the following command to grant executable permissions on the JAR file:

   chmod 777 kafka-migration-assessment.jar

4. Run the following command to export the metadata:

   java -jar kafka-migration-assessment.jar MigrationFromZk --sourceZkConnect <yourHost:yourPort> --sourceBootstrapServers <yourHost:yourPort> --targetDirectory <yourDirectoryName> --fileName <yourFileName> --commit

   Parameter	Description	Example
   sourceZkConnect	The IP address and port of the ZooKeeper service for your self-managed Kafka cluster. If not specified, the tool attempts automatic discovery.	192.168.XX.XX:2181
   sourceBootstrapServers	The IP address and port of your self-managed Kafka cluster. If not specified, the tool attempts automatic discovery.	192.168.XX.XX:9092
   targetDirectory	The directory for the exported metadata file. Defaults to the current directory.	../home/
   fileName	The name of the exported metadata file. Default: kafka-metadata-export.json.	metadata.json
   commit	Required to run the export command.	commit
   installTsar	Whether to automatically install Tsar. Not installed by default. Tsar provides more accurate information about the machine's specifications, storage usage, traffic, and Kafka cluster configuration. However, installation can be time-consuming and may fail due to environmental differences.	None
   evaluate	Whether to retrieve the machine's specifications, storage usage, traffic, and Kafka cluster configuration. This information helps recommend a suitable ApsaraMQ for Kafka instance specification. Default: true. Set to false if specification evaluation is not needed.	None

   A successful execution generates a JSON file in the specified directory.

5. Locate the exported metadata file in the configured directory and download it to your local machine.

Step 3: Create a metadata import task

1. Log on to the ApsaraMQ for Kafka console. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.

2. In the left-side navigation pane, click Migration. On the page that appears, click the Metadata Import tab.

3. On the Metadata Import tab, click Create Task.

4. In the Create Metadata Import Task panel, configure the parameters and click Create.

   1. On the Create Task tab, set the Task Name and Destination Instance, and then upload the metadata file that you obtained in Step 2: Export metadata.

   2. On the Edit Topic tab, you can perform the following operations:

      • To add a topic, click Add Topic and enter the Name, Description, Partitions, and Message Type for the topic.

      • To edit a topic, click Modify in the Actions column and change the Description, Partitions, Message Type, or Log Cleanup Policy.

      • To delete a topic, click Delete in the Actions column, and then click OK in the Note dialog box.

   3. On the Edit Group tab, you can perform the following operations:

      • To add a group, click Add Group and enter the Group ID and Description.

      • To edit a group, click Modify in the Actions column and change the Description.

      • To delete a group, click Delete in the Actions column, and then click OK in the Note dialog box.

   After the task is created, you can view its progress on the Migration page. The Topic Migration Progress and Group Migration Progress columns show the status. To view more information, click Details in the Actions column.

Step 4: Verify migration results

1. Log on to the ApsaraMQ for Kafka console. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.

2. In the left-side navigation pane, click Instances.

3. On the Instances page, click the name of the destination instance.

4. Verify the imported resources.

   • In the left-side navigation pane, click Topics. On the Topics page, verify that the imported topics appear on the page.

   • In the left-side navigation pane, click Groups. On the Groups page, verify that the imported groups appear on the page.