ApsaraMQ for Kafka:Use a migration tool

Implement migration

1. Log on to the server of your self-managed Apache Kafka cluster.

2. Download and install JDK 8 or 11. For more information, see Java Downloads.

3. Download the migration tool: kafka-migration-assessment.jar.

4. Migrate topics and consumer groups using the following methods.

   Migrate topics

   1. In the directory where the migration tool is located, run the following command to perform a precheck on the topics to be migrated:

      java -jar kafka-migration-assessment.jar TopicMigrationFromZk  \
      --sourceZkConnect 192.168.XX.XX  \
      --destAk <yourdestAccessKeyId>  \
      --destSk <yourdestAccessKeySecret>  \
      --destRegionId <yourdestRegionId>  \
      --destInstanceId <yourdestInstanceId>

      Parameter	Description
      sourceZkConnect	The IP address of the source ZooKeeper cluster.
      destAk	The AccessKey ID of the Alibaba Cloud account to which the destination ApsaraMQ for Kafka instance belongs.
      destSk	The AccessKey Secret of the Alibaba Cloud account to which the destination ApsaraMQ for Kafka instance belongs.
      destRegionId	The region ID of the destination ApsaraMQ for Kafka instance.
      destInstanceId	The ID of the destination ApsaraMQ for Kafka instance.

      Sample returned result to be confirmed:

      13:40:08 INFO - Begin to migrate topics:[test]
      13:40:08 INFO - Total topic number:1
      13:40:08 INFO - Will create topic:test, isCompactTopic:false, partition number:1

   2. Run the following command to migrate topics.

      java -jar kafka-migration-assessment.jar TopicMigrationFromZk  \
      --sourceZkConnect 192.168.XX.XX  \
      --destAk <yourdestAccessKeyId>  \
      --destSk <yourdestAccessKeySecret>  \
      --destRegionId <yourdestRegionId>  \
      --destInstanceId <yourdestInstanceId>  \
      --commit

      Parameter	Description
      commit	Commits the migration task

      Sample returned result after committing the migration task:

      13:51:12 INFO - Begin to migrate topics:[test]
      13:51:12 INFO - Total topic number:1
      13:51:13 INFO - cmd=TopicMigrationFromZk, request=null, response={"code":200,"requestId":"7F76C7D7-AAB5-4E29-B49B-CD6F1E0F508B","success":true,"message":"operation success"}
      13:51:13 INFO - TopicCreate success, topic=test, partition number=1, isCompactTopic=false

   Migrate consumer groups

   1. Create a configuration file named kafka.properties.

      The kafka.properties file is used to initialize a Kafka consumer to obtain the consumer offsets of the self-managed Apache Kafka cluster. The file contains the following content:

      ## The endpoint.
      bootstrap.servers=localhost:9092
      
      ## The group ID. To ensure that consumption starts from the first message, the group cannot contain consumer offset information.
      group.id=XXX
      
      ## If no security settings are required, you do not need to configure the following parameters.
      
      ## The SASL-based authentication.
      #sasl.mechanism=PLAIN
      
      ## The access protocol.
      #security.protocol=SASL_SSL
      
      ## The path of the Secure Sockets Layer (SSL) root certificate.
      #ssl.truststore.location=/Users/***/Documents/code/aliware-kafka-demos/main/resources/kafka.client.truststore.jks
      
      ## The SSL password.
      #ssl.truststore.password=***
      
      ## The SASL path.
      #java.security.auth.login.config=/Users/***/kafka-java-demo/vpc-ssl/src/main/resources/kafka_client_jaas.conf

   2. In the directory where the migration tool is located, run the following command to perform a precheck on the consumer groups to be migrated:

      java -jar kafka-migration-assessment.jar ConsumerGroupMigrationFromTopic  \
      --propertiesPath /usr/local/kafka_2.12-2.4.0/config/kafka.properties  \
      --destAk <yourAccessKeyId>  \
      --destSk <yourAccessKeySecret>  \
      --destRegionId <yourRegionId>  \
      --destInstanceId <yourInstanceId>

      Parameter	Description
      propertiesPath	The path of the kafka.properties configuration file.
      destAk	The AccessKey ID of the Alibaba Cloud account to which the destination ApsaraMQ for Kafka instance belongs.
      destSk	The AccessKey Secret of the Alibaba Cloud account to which the destination ApsaraMQ for Kafka instance belongs.
      destRegionId	The region ID of the destination ApsaraMQ for Kafka instance.
      destInstanceId	The ID of the destination ApsaraMQ for Kafka instance.

      Sample returned result to be confirmed:

      15:29:45 INFO - Will create consumer groups:[XXX, test-consumer-group]

   3. Run the following command to migrate consumer groups:

      java -jar kafka-migration-assessment.jar ConsumerGroupMigrationFromTopic  \
      --propertiesPath /usr/local/kafka_2.12-2.4.0/config/kafka.properties  \
      --destAk <yourAccessKeyId>  \
      --destSk <yourAccessKeySecret>  \
      --destRegionId <yourRegionId>  \
      --destInstanceId <yourInstanceId>  \
      --commit

      Parameter	Description
      commit	Commit the migration task.

      Sample returned result after committing the migration task:

      15:35:51 INFO - cmd=ConsumerGroupMigrationFromTopic, request=null, response={"code":200,"requestId":"C9797848-FD4C-411F-966D-0D4AB5D12F55","success":true,"message":"operation success"}
      15:35:51 INFO - ConsumerCreate success, consumer group=XXX
      15:35:57 INFO - cmd=ConsumerGroupMigrationFromTopic, request=null, response={"code":200,"requestId":"3BCFDBF2-3CD9-4D48-92C3-385C8DBB9709","success":true,"message":"operation success"}
      15:35:57 INFO - ConsumerCreate success, consumer group=test-consumer-group

View the migration progress

1. Log on to the ApsaraMQ for Kafka console. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.

2. In the left-side navigation pane, click Migration. On the page that appears, click the Metadata Import tab.

3. On the Metadata Import tab, find the destination ApsaraMQ for Kafka instance in your migration task and view the migration progress of the topics and consumer groups.

Verify the results

1. Log on to the ApsaraMQ for Kafka console. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.

2. In the left-side navigation pane, click Instances.

3. On the Instances page, click the name of the instance that you want to manage.

4. View the topics and groups on the instance.

   • In the left-side navigation pane, click Topics. On the page that appears, view the created topics on the instance.

   • In the left-side navigation pane, click Groups. On the page that appears, view the created groups on the instance.

Prerequisites

• The migration tool MirrorMaker is downloaded.

• Topics are migrated.

Usage notes

• Topic names must be consistent.

• The number of partitions can be different.

• Data in the same partition may not remain in the same partition after migration.

• By default, messages with the same key are distributed in the same partition.

• When a node fails, normal messages may be out of order while partitionally ordered messages can retain their order.

• If both the self-managed Apache Kafka cluster and the ApsaraMQ for Kafka instance use password-based authentication and the two passwords are inconsistent, migration is not supported.

Implement migration

You can choose to access from the Internet or a VPC.

Verify the results

You can use one of the following methods to verify whether MirrorMaker is running as expected:

• Check the consumer progress of the self-managed Apache Kafka cluster using the kafka-consumer-groups.sh command.

  bin/kafka-consumer-groups.sh --new-consumer --describe --bootstrap-server endpoint of the self-managed Apache Kafka cluster --group test-consumer-group

• Send messages to the self-managed Apache Kafka cluster. In the ApsaraMQ for Kafka console, check the partition status of the topic, and check whether the total number of messages in the current broker is correct. You can view the content of a message in the ApsaraMQ for Kafka console. For more information, see Query messages.