Migrate topics from a self-managed Kafka cluster to Message Queue for Apache Kafka - Message Queue for Apache Kafka

This topic describes how to use the migration tool provided by Message Queue for Apache Kafka to migrate topics from a self-managed Kafka cluster to a Message Queue for Apache Kafka instance.

Prerequisites

Make sure that the following operations are complete:

Java Development Kit (JDK) 8 is downloaded and installed. For more information, see Java Downloads.
The migration tool is downloaded and installed. If the migration tool is not downloaded, click Kafka-migration-assessment.jar.zip to download the migration tool.
A Message Queue for Apache Kafka instance is purchased and deployed.
- For information about how to purchase an instance and deploy the instance in a virtual private cloud (VPC), see Purchase and deploy an instance that allows access from a VPC.
- For information about how to purchase an instance that can be accessed over the Internet and a VPC and deploy the instance, see Purchase and deploy an instance that allows access from the Internet and a VPC.

Procedure

Open your CLI.
Run the cd command to go to the directory where the migration tool is stored.

Run the following command to check the topics that you want to migrate:

java -jar kafka-migration-assessment.jar TopicMigrationFromZk --sourceZkConnect 192.168.XX.XX
                           --destAk <yourdestAccessKeyId> --destSk <yourdestAccessKeySecret> --destRegionId <yourdestRegionId>
                           --destInstanceId <yourdestInstanceId>


Parameter	Description
sourceZkConnect	The IP address of the source Apache Kafka cluster in ZooKeeper.
destAk	The AccessKey ID of the Alibaba Cloud account to which the destination Message Queue for Apache Kafka instance belongs.
destSk	The AccessKey secret of the Alibaba Cloud account to which the destination Message Queue for Apache Kafka instance belongs.
destRegionId	The ID of the region where the destination Message Queue for Apache Kafka instance is deployed.
destInstanceId	The ID of the destination Message Queue for Apache Kafka instance.

Check the information that is returned. The returned information describes the topics that are migrated.

13:40:08 INFO - Begin to migrate topics:[test]
13:40:08 INFO - Total topic number:1
13:40:08 INFO - Will create topic:test, isCompactTopic:false, partition number:1

Run the following command to commit the migration task:

java -jar kafka-migration-assessment.jar TopicMigrationFromZk --sourceZkConnect 192.168.XX.XX
                           --destAk <yourAccessKeyId> --destSk <yourAccessKeySecret> --destRegionId <yourRegionID>
                           --destInstanceId <yourInstanceId> --commit


Parameter	Description
commit	Commits the migration task.

After you commit the migration task, the following information is returned:

13:51:12 INFO - Begin to migrate topics:[test]
13:51:12 INFO - Total topic number:1
13:51:13 INFO - cmd=TopicMigrationFromZk, request=null, response={"code":200,"requestId":"7F76C7D7-AAB5-4E29-B49B-CD6F1E0F508B","success":true,"message":"operation success"}
13:51:13 INFO - TopicCreate success, topic=test, partition number=1, isCompactTopic=false

Check whether the required topic is migrated.
1. Log on to the Message Queue for Apache Kafka console.
2. In the Resource Distribution section of the Overview page, select the region where your instance resides.
3. On the Instances page, click the name of the instance that you want to manage.
4. In the left-side navigation pane, click Topics.
5. On the Topics page, check whether the required topic is displayed.