This topic describes how to use the migration tool provided by Message Queue for Apache Kafka to migrate a group from a self-managed Kafka cluster to a Message Queue for Apache Kafka instance.Group

Prerequisites

Make sure that the following operations are complete:

Procedure

  1. Open your CLI.
  2. Run the cd command to go to the directory in which the migration tool is stored.
  3. Create a configuration file named kafka.properties.
    The system defines subscription relationships and obtains the consumer offset of the original consumer group based on the settings that you configured in the kafka.properties file. The configuration file contains the following information:
    ## The endpoint. 
    bootstrap.servers=localhost:9092
    
    ## The ID of the group.Group Make sure that the group does not contain consumer offset information. If the group contains consumer offset information, the group may not consume messages from the first message that needs to be consumed.Group 
    group.id=XXX
    
    ## If you do not require certificate authorization, you do not need to configure the following parameters. 
    
    ## The Simple Authentication and Security Layer (SASL) mechanism. 
    #sasl.mechanism=PLAIN
    
    ## The access protocol. 
    #security.protocol=SASL_SSL
    
    ## The path of the root Secure Sockets Layer (SSL) certificate. 
    #ssl.truststore.location=/Users/***/Documents/code/aliware-kafka-demos/main/resources/kafka.client.truststore.jks
    
    ## The password of the root SSL certificate. 
    #ssl.truststore.password=***
    
    ## The path of the configuration file that you want to use for SASL. 
    #java.security.auth.login.config=/Users/***/kafka-java-demo/vpc-ssl/src/main/resources/kafka_client_jaas.conf
  4. Run the following command to check the information about the group that you want to migrate:Group
    java -jar kafka-migration-assessment.jar ConsumerGroupMigrationFromTopic --propertiesPath /usr/local/kafka_2.12-2.4.0/config/kafka.properties --destAk <yourAccessKeyId> --destSk <yourAccessKeySecret> --destRegionId <yourRegionId> --destInstanceId <yourInstanceId>
    Parameter Description
    propertiesPath The path of the kafka.properties configuration file.
    destAk The AccessKey ID of the Alibaba Cloud account to which the destination Message Queue for Apache Kafka instance belongs.
    destSk The AccessKey secret of the Alibaba Cloud account to which the destination Message Queue for Apache Kafka instance belongs.
    destRegionId The ID of the region where the destination Message Queue for Apache Kafka instance is deployed.
    destInstanceId The ID of the destination Message Queue for Apache Kafka instance.
    Check the information that is returned. The returned information describes the group that is migrated.
    15:29:45 INFO - Will create consumer groups:[XXX, test-consumer-group]
  5. Run the following command to commit the migration task:Group
    java -jar kafka-migration-assessment.jar ConsumerGroupMigrationFromTopic --propertiesPath /usr/local/kafka_2.12-2.4.0/config/kafka.properties --destAk LTAI4FwQ5aK1mFYCspJ1**** --destSk wvDxjjRQ1tHPiL0oj7Y2Z7WDNk**** --destRegionId cn-hangzhou --destInstanceId alikafka_pre-cn-v0h1cng0**** --commit
    Parameter Description
    commit Commits the migration task.
    After you commit the migration task, the following information is returned:
    15:35:51 INFO - cmd=ConsumerGroupMigrationFromTopic, request=null, response={"code":200,"requestId":"C9797848-FD4C-411F-966D-0D4AB5D12F55","success":true,"message":"operation success"}
    15:35:51 INFO - ConsumerCreate success, consumer group=XXX
    15:35:57 INFO - cmd=ConsumerGroupMigrationFromTopic, request=null, response={"code":200,"requestId":"3BCFDBF2-3CD9-4D48-92C3-385C8DBB9709","success":true,"message":"operation success"}
    15:35:57 INFO - ConsumerCreate success, consumer group=test-consumer-group

    After you commit the task to migrate the specified group to the Message Queue for Apache Kafka instance, you can log on to the Message Queue for Apache Kafka console and view information about the migration task on the Migration to Cloud page.