This topic describes how to use a metadata migration tool provided by Message Queue for Apache Kafka to migrate consumer group metadata from a self-managed Apache Kafka cluster to a Message Queue for Apache Kafka instance.

Prerequisites

Before you migrate consumer group metadata from a self-managed Apache Kafka cluster to Message Queue for Apache Kafka, make sure that the following operations are completed:
Notice
  • After the migration, the corresponding consumer group in the source self-managed Apache Kafka cluster is not deleted. Instead, a new consumer group with the same configuration is created in the destination Message Queue for Apache Kafka instance.
  • Only the configuration of the consumer group in the source Apache Kafka cluster is migrated. The topics to which the consumer group subscribes and related consumer offsets are not migrated.

Procedure

  1. Start the command-line tool.
  2. Run the cd command to switch to the directory where the migration tool is located.
  3. Create a configuration file named kafka.properties.
    The kafka.properties file is used to create an Apache Kafka consumer to retrieve consumer offset information from the self-managed Apache Kafka cluster. The configuration file contains the following content:
    ## The endpoint. 
    bootstrap.servers=localhost:9092
    
    ## The consumer group, which does not contain consumer offset information so that consumption can start from the first message. 
    group.id=XXX
    
    ## You can skip the following configuration if security configuration is unavailable. 
    
    ## The Simple Authentication and Security Layer (SASL) mechanism. 
    #sasl.mechanism=PLAIN
    
    ## The access protocol. 
    #security.protocol=SASL_SSL
    
    ## The path of the root SSL certificate. 
    #ssl.truststore.location=/Users/***/Documents/code/aliware-kafka-demos/main/resources/kafka.client.truststore.jks
    
    ## The password of the root SSL certificate. 
    #ssl.truststore.password=***
    
    ## The path to the configuration file for SASL. 
    #java.security.auth.login.config=/Users/***/kafka-java-demo/vpc-ssl/src/main/resources/kafka_client_jaas.conf
  4. Run the following command to confirm the consumer group metadata to be migrated:
    java -jar kafka-migration.jar ConsumerGroupMigrationFromTopic --propertiesPath /usr/local/kafka_2.12-2.4.0/config/kafka.properties --destAk <yourAccessKeyId> --destSk <yourAccessKeySecret> --destRegionId <yourRegionId> --destInstanceId <yourInstanceId>
    Parameter Description
    propertiesPath The path of the kafka.properties configuration file.
    destAk The AccessKey ID of the Alibaba Cloud account to which the destination Message Queue for Apache Kafka instance belongs.
    destSk The AccessKey secret of the Alibaba Cloud account to which the destination Message Queue for Apache Kafka instance belongs.
    destRegionId The ID of the region where the destination Message Queue for Apache Kafka instance is located.
    destInstanceId The ID of the destination Message Queue for Apache Kafka instance.
    The following code provides an example of the output to be confirmed:
    15:29:45 INFO - Will create consumer groups:[XXX, test-consumer-group]
  5. Run the following command to commit the consumer group metadata to be migrated:
    java -jar kafka-migration.jar ConsumerGroupMigrationFromTopic --propertiesPath /usr/local/kafka_2.12-2.4.0/config/kafka.properties --destAk LTAI4FwQ5aK1mFYCspJ1h*** --destSk wvDxjjRQ1tHPiL0oj7Y2Z7WDNkS*** --destRegionId cn-hangzhou --destInstanceId alikafka_pre-cn-v0h1cng00*** --commit
    Parameter Description
    commit Commits the consumer group metadata to be migrated.
    The following code provides an example of the output generated after the consumer group metadata to be migrated is committed:
    15:35:51 INFO - cmd=ConsumerGroupMigrationFromTopic, request=null, response={"code":200,"requestId":"C9797848-FD4C-411F-966D-0D4AB5D12F55","success":true,"message":"operation success"}
    15:35:51 INFO - ConsumerCreate success, consumer group=XXX
    15:35:57 INFO - cmd=ConsumerGroupMigrationFromTopic, request=null, response={"code":200,"requestId":"3BCFDBF2-3CD9-4D48-92C3-385C8DBB9709","success":true,"message":"operation success"}
    15:35:57 INFO - ConsumerCreate success, consumer group=test-consumer-group
  6. Check whether the required consumer group is migrated.
    1. Log on to the Message Queue for Apache Kafka console.
    2. In the Resource Distribution section of the Overview page, select the region where your instance resides.
    3. On the Instances page, click the name of the instance that you want to manage.
    4. In the left-side navigation pane, click Groups.
    5. On the Groups page, check whether the required consumer group is displayed.