Migrate data to the cloud

Precautions

• The topic names must be consistent.
• The numbers of partitions can be inconsistent.
• Data in the same partition is not certain to be migrated to the same partition.
• By default, messages with the same key are distributed in the same partition.
• Normal messages may be out of order when the instance fails, whereas partitionally ordered messages remain in order when the instance fails.

Access from a VPC

1. Configure the consumer.properties file.

   ## The endpoint of the user-created Kafka cluster.
   bootstrap.servers=XXX.XXX.XXX.XXX:9092
   
   ## The consumer policy for distributing messages to partitions.
   partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor
   
   ## The name of the consumer group.
   group.id=test-consumer-group

2. Configure the producer.properties file.

   ## The default endpoint of the Message Queue for Apache Kafka instance, which can be obtained in the Message Queue for Apache Kafka console.
   bootstrap.servers=XXX.XXX.XXX.XXX:9092
   
   ## The data compression method.
   compression.type=none                                

3. Run the following command to start the migration process:

   sh bin/kafka-mirror-maker.sh --consumer.config config/consumer.properties --producer.config config/producer.properties --whitelist topicName

Access from the Internet

1. Download kafka.client.truststore.jks.
2. Configure the kakfa_client_jaas.conf file.

   KafkaClient {
      org.apache.kafka.common.security.plain.PlainLoginModule required
      username="your username"
      password="your password";
   };

3. Configure the consumer.properties file.

   ## The endpoint of the user-created Kafka cluster.
   bootstrap.servers=XXX.XXX.XXX.XXX:9092
   
   ## The consumer policy for distributing messages to partitions.
   partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor
   
   ## The name of the consumer group.
   group.id=test-consumer-group

4. Configure the producer.properties file.

   ## The Secure Sockets Layer (SSL) endpoint of the Message Queue for Apache Kafka instance, which can be obtained in the Message Queue for Apache Kafka console.
   bootstrap.servers=XXX.XXX.XXX.XXX:9093
   
   ## The data compression method.
   compression.type=none
   
   ## The truststore (Use the file downloaded in Step 1).
   ssl.truststore.location=kafka.client.truststore.jks
   ssl.truststore.password=KafkaOnsClient
   security.protocol=SASL_SSL
   sasl.mechanism=PLAIN
   
   ## If you use Message Queue for Apache Kafka 2.x, you must set the following parameter in Simple Authentication and Security Layer (SASL) access mode. If you use Message Queue for Apache Kafka earlier than v2.x, you do not need to set it.
   ssl.endpoint.identification.algorithm=

5. Configure the java.security.auth.login.config file.

   export KAFKA_OPTS="-Djava.security.auth.login.config=kafka_client_jaas.conf"                              

6. Run the following command to start the migration process:

   sh bin/kafka-mirror-maker.sh --consumer.config config/consumer.properties --producer.config config/producer.properties --whitelist topicName

Verify the result

You can check whether MirrorMaker runs by using one of the following methods:

• Run kafka-consumer-groups.sh to view the consumption progress of the user-created cluster.

  bin/kafka-consumer-groups.sh --new-consumer --describe --bootstrap-server user-created cluster endpoint --group test-consumer-group

• Send messages to the user-created cluster. In the Message Queue for Apache Kafka console, check the partition status of the topic, and check whether the total number of messages in the current broker is correct. You can view the specific content of a message in the Message Queue for Apache Kafka console. For more information, see Message query.