ApsaraMQ for Kafka:Import metadata from a self-managed Apache Kafka cluster

What gets migrated

How it works

The metadata import follows a four-step process:

1. (Optional) Evaluate specifications -- Get recommended instance specifications based on your source cluster's workload.

2. Export metadata -- Run a JAR tool on a node in your self-managed cluster. The tool connects to ZooKeeper, reads topic and consumer group configurations, and writes them to a JSON file.

3. Create a metadata import task -- Upload the JSON file to the ApsaraMQ for Kafka console and review topics and consumer groups before creating them on the destination instance.

4. Verify the result -- Confirm that topics and consumer groups appear on the destination instance.

Prerequisites

Before you begin, make sure that you have:

• Java Development Kit (JDK) 8 installed (download JDK 8)

• An ApsaraMQ for Kafka instance purchased and deployed:

  • Purchase and deploy a VPC-connected instance

  • Purchase and deploy an Internet- and VPC-connected instance

Limitations

• Metadata import is not supported for serverless ApsaraMQ for Kafka instances.

Evaluate specifications (optional)

Use the specification evaluation feature to get recommended instance specifications based on your self-managed cluster's workload.

1. Log on to the ApsaraMQ for Kafka console. In the Resource Distribution section of the Overview page, select the region of your destination instance.

2. In the left-side navigation pane, click Migration. Click the Metadata Import tab.

3. Click Evaluation.

4. In the Evaluation panel, enter your source cluster's workload information:

   Parameter	Description	Required	Example
   Major Version	Apache Kafka version in your self-managed cluster	Yes	0.10.x
   Migration from IDC	Whether the source cluster runs in an Internet data center (IDC)	Yes	Yes
   Maximum Cluster Traffic	Peak message production or consumption throughput, in MB/s	Yes	20
   Public Network Traffic	Required internet bandwidth, in MB/s. Enter 0 if internet access is not needed	Yes	0
   SSD	Whether the source cluster uses solid-state drives (SSDs)	Yes	Yes
   Disk Capacity	Total disk capacity, in GB	Yes	500
   Partitions	Total number of partitions across all topics to migrate	Yes	800
   Replicas	Number of replicas per topic	Yes	3 Replicas
   Scenario	Workload type for message production and consumption	Yes	Business Message
   Response Latency	Latency requirements for message production and consumption	Yes	No Special Requirements

5. Click OK. The Recommended Instance Specifications section displays the recommended edition, traffic specification, and disk size for your destination instance.

Export the metadata

Run the metadata export tool on a node in your self-managed Apache Kafka cluster to generate a JSON file containing topic and consumer group configurations.

1. Download the metadata export tool: kafka-migration-assessment.jar.

2. Upload the JAR file to a node in your self-managed Kafka cluster.

3. Grant execute permission on the JAR file:

      chmod 777 kafka-migration-assessment.jar

4. Run the export command: Replace the placeholders with your values:

   Parameter	Description	Required	Default	Example
   sourceZkConnect	IP address and port of the ZooKeeper node for your self-managed cluster. If omitted, the tool auto-detects this value	No	Auto-detected	192.168.XX.XX:2181
   sourceBootstrapServers	IP address and port of a Kafka broker in your self-managed cluster. If omitted, the tool auto-detects this value	No	Auto-detected	192.168.XX.XX:9092
   targetDirectory	Directory to write the output JSON file	No	Current directory	../home/
   fileName	Name of the output JSON file	No	kafka-metadata-export.json	metadata.json
   commit	Commits the task	Yes	--	--
   installTsar	Installs Taobao System Activity Reporter (Tsar) to collect machine specifications and resource usage. Installation may take a long time and can fail in incompatible environments	No	Not installed	--
   evaluate	Collects machine specifications and resource usage for specification evaluation. Set to false to skip	No	true	false

      java -jar kafka-migration-assessment.jar MigrationFromZk \
        --sourceZkConnect <zookeeper-host:port> \
        --sourceBootstrapServers <kafka-host:port> \
        --targetDirectory <output-directory> \
        --fileName <output-filename> \
        --commit

5. Verify that the JSON file exists in the specified output directory.

   Note

   Download the file to your local machine for uploading to the ApsaraMQ for Kafka console in the next step.

Create a metadata import task

Upload the exported JSON file and configure the topics and consumer groups to create on the destination ApsaraMQ for Kafka instance.

1. Log on to the ApsaraMQ for Kafka console. In the Resource Distribution section of the Overview page, select the region of your destination instance.

2. In the left-side navigation pane, click Migration. Click the Metadata Import tab.

3. Click Create Task.

4. In the Create Metadata Import Task panel, complete the following steps: Create Task Configure Task Name and Destination Instance, then upload the metadata JSON file exported in Export the metadata. Edit Topic Review the imported topics and adjust as needed: Edit Group Review the imported consumer groups and adjust as needed:

   • Add a group: Click Add Group and configure the Group ID and Description parameters.

   • Modify a group: Click Modify in the Actions column to update the Description.

   • Delete a group: Click Delete in the Actions column, then click OK in the confirmation dialog.

5. Click Create to start the metadata import task. After the task is created, the Metadata Import tab shows the Topic Migration Progress and Group Migration Progress for each task. Click Details in the Actions column to view the full task details.

Verify the migration result

Confirm that the topics and consumer groups were created on the destination instance.

1. Log on to the ApsaraMQ for Kafka console. In the Resource Distribution section of the Overview page, select the region of your destination instance.

2. In the left-side navigation pane, click Instances. Click the name of the destination instance.

3. Verify the topics: In the left-side navigation pane, click Topics. Confirm that all expected topics are listed.

4. Verify the consumer groups: In the left-side navigation pane, click Groups. Confirm that all expected consumer groups are listed.