When migrating a self-managed Apache Kafka cluster that lacks internet access, you need to transfer topic and consumer group configurations to your managed instance before cutting over producers and consumers. Metadata import handles this by exporting configurations from your self-managed cluster as a JSON file and importing them into an ApsaraMQ for Kafka instance -- without migrating message data.
Choose the right migration method
ApsaraMQ for Kafka provides two migration methods. Choose based on your cluster's network connectivity and whether you need to migrate message data.
| Scenario | Method | What it migrates |
|---|---|---|
| Cluster without internet access, metadata only | Metadata import (this guide) | Topic and consumer group configurations |
| Cluster with internet access, or metadata + messages | Cloud migration | Metadata and message data |
Serverless ApsaraMQ for Kafka instances do not support metadata import tasks.
What metadata is imported
Metadata import copies the following configurations from your self-managed cluster to an ApsaraMQ for Kafka instance:
| Metadata type | Imported fields |
|---|---|
| Topics | Name, partition count, message type, log cleanup policy |
| Consumer groups | Group ID, description |
Metadata import does not transfer message data. To migrate both metadata and message data, use the cloud migration feature instead.
How it works
The migration follows five sequential steps:
Evaluate specifications -- Assess the ApsaraMQ for Kafka instance specs required for your workload.
Purchase an instance -- Deploy an ApsaraMQ for Kafka instance based on the evaluation.
Export metadata -- Run the export tool on your self-managed cluster to generate a JSON metadata file.
Create an import task -- Upload the metadata file in the ApsaraMQ for Kafka console and configure topics and groups.
Verify the result -- Confirm that topics and groups were created on the target instance.
Prerequisites
Before you begin, make sure that you have:
A self-managed Apache Kafka cluster with accessible ZooKeeper and bootstrap server endpoints
JDK 8 or 11 installed on the cluster server (download JDK)
An ApsaraMQ for Kafka instance (non-serverless) deployed in your target region
Step 1: Evaluate specifications
Use the specification evaluation feature to determine the ApsaraMQ for Kafka instance specs required for your migration. The evaluation considers your cluster's traffic, disk capacity, and disk type.
For details, see Evaluate specifications.
Step 2: Purchase and deploy an instance
Purchase an ApsaraMQ for Kafka instance based on the evaluated specifications. Choose a deployment option based on your networking requirements:
Internet + VPC access: See Purchase and deploy an Internet- and VPC-connected instance.
VPC-only access: See Purchase and deploy a VPC-connected instance.
Step 3: Export metadata
Run the metadata export tool on your self-managed Apache Kafka cluster to generate a JSON file containing topic and consumer group configurations.
Download the export tool: kafka-migration-assessment.jar.
Grant execute permission on the JAR file:
chmod 777 kafka-migration-assessment.jarRun the export command: Replace the placeholders with your actual values:
Parameter Description Default Example sourceZkConnectZooKeeper endpoint of your self-managed cluster Auto-detected 192.168.XX.XX:2181sourceBootstrapServersBootstrap server endpoint of your self-managed cluster Auto-detected 192.168.XX.XX:9092targetDirectoryDirectory to save the exported metadata file Current directory ../home/fileNameName of the exported metadata file kafka-metadata-export.jsonmetadata.jsoncommitCommits the export task. Required. -- -- Optional parameters:
Parameter Description Default evaluateCollect machine specs and recent usage data for specification evaluation. Set to falseto skip.trueinstallTsarInstall Taobao System Activity Reporter (Tsar) for detailed system metrics. Installation may take time and can fail in incompatible environments. Not installed java -jar kafka-migration-assessment.jar MigrationFromZk \ --sourceZkConnect <zookeeper-ip>:<port> \ --sourceBootstrapServers <kafka-ip>:<port> \ --targetDirectory <output-directory> \ --fileName <output-filename> \ --commitVerify the export by checking that a JSON file exists in the output directory:
ls <output-directory>/<output-filename>(Optional) Download the metadata file to your local machine if you plan to upload it from a different device.
Step 4: Create a metadata import task
Upload the exported metadata file in the ApsaraMQ for Kafka console and review the topics and groups before importing.
Log on to the ApsaraMQ for Kafka console.
In the Resource Distribution section of the Overview page, select the region of your target instance.
In the left-side navigation pane, click Migration, then click the Metadata Import tab.
Click Create Task.
In the Create Metadata Import Task panel, complete the following steps: a. Create Task Configure the Task Name and Destination Instance, then upload the metadata JSON file exported in Step 3. b. Edit Topic Review the imported topics and adjust as needed: c. Edit Group Review the imported consumer groups and adjust as needed:
Action How Configurable fields Add a topic Click Add Topic Name, Description, Partitions, Message Type Modify a topic Click Modify in the Actions column Description, Partitions, Message Type, Log Cleanup Policy Delete a topic Click Delete in the Actions column, then click OK in the Note dialog -- Action How Configurable fields Add a group Click Add Group Group ID, Description Modify a group Click Modify in the Actions column Description Delete a group Click Delete in the Actions column, then click OK in the Note dialog -- Click Create to start the import task.
After the task is created, the Metadata Import tab displays the migration progress. Check the Topic Migration Progress and Group Migration Progress columns. Click Details in the Actions column to view task details.
Step 5: Verify the migration result
Confirm that topics and consumer groups were created on the target ApsaraMQ for Kafka instance.
Log on to the ApsaraMQ for Kafka console.
In the Resource Distribution section of the Overview page, select the region of your target instance.
In the left-side navigation pane, click Instances. Click the name of the target instance.
Verify topics:
In the left-side navigation pane, click Topics.
Confirm that all expected topics are listed with the correct partition counts and message types.
Verify consumer groups:
In the left-side navigation pane, click Groups.
Confirm that all expected consumer groups are listed.
Tip: Compare the topic and group counts against the totals shown in the Topic Migration Progress and Group Migration Progress columns on the Metadata Import tab to make sure all items were migrated.
Cut over producers and consumers
After metadata import completes, cut over your producers and consumers in the following order to avoid message loss:
Start new consumers on ApsaraMQ for Kafka. Set up consumer groups on the ApsaraMQ for Kafka instance and start consuming messages.
Switch producers to ApsaraMQ for Kafka. Redirect producers to send messages to the ApsaraMQ for Kafka instance. Shut down the original producers.
Drain the self-managed cluster. Keep the original consumer groups running on the self-managed cluster until all remaining messages are consumed. Then shut down the original consumer groups and decommission the self-managed cluster.
Follow this exact order -- consumers first, then producers, then drain. Starting producers before consumers on the new instance risks unprocessed message buildup. Shutting down original consumers before draining risks message loss.