ApsaraMQ for Kafka:Migrate data to the cloud

Precautions

• The topic names of the source and destination clusters must be consistent.
• The number of partitions in the source and destination clusters can be different.
• Data that resides in a partition of the source cluster may be distributed to different partitions in the destination cluster.
• By default, messages with the same key are distributed to the same partition.
• Normal messages may become out of order when the cluster fails, whereas partitionally ordered messages remain in order.

Access from a VPC

1. Configure the consumer.properties file.

   ## The endpoint of the user-created Kafka cluster.
   bootstrap.servers=XXX.XXX.XXX.XXX:9092
   
   ## The consumer policy for distributing messages to partitions.
   partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor
   
   ## The name of the group.
   group.id=test-consumer-group

2. Configure the producer.properties file.

   ## The default endpoint of the ApsaraMQ for Kafka cluster, which can be obtained in the ApsaraMQ for Kafka console.
   bootstrap.servers=XXX.XXX.XXX.XXX:9092
   
   ## The data compression method.
   compression.type=none                                

3. Run the following command to start the migration process:

   sh bin/kafka-mirror-maker.sh --consumer.config config/consumer.properties --producer.config config/producer.properties --whitelist topicName

Access from the Internet

1. Download kafka.client.truststore.jks.
2. Configure the kafka_client_jaas.conf file.

   KafkaClient {
      org.apache.kafka.common.security.plain.PlainLoginModule required
      username="your username"
      password="your password";
   };

3. Configure the consumer.properties file.

   ## The endpoint of the user-created Kafka cluster.
   bootstrap.servers=XXX.XXX.XXX.XXX:9092
   
   ## The consumer policy for distributing messages to partitions.
   partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor
   
   ## The name of the group.
   group.id=test-consumer-group

4. Configure the producer.properties file.

   ## The SSL endpoint of the ApsaraMQ for Kafka cluster, which can be obtained in the ApsaraMQ for Kafka console.
   bootstrap.servers=XXX.XXX.XXX.XXX:9093
   
   ## The data compression method.
   compression.type=none
   
   ## The truststore (Use the file downloaded in Step 1).
   ssl.truststore.location=kafka.client.truststore.jks
   ssl.truststore.password=KafkaOnsClient
   security.protocol=SASL_SSL
   sasl.mechanism=PLAIN
   
   ## To use Simple Authentication and Security Layer (SASL) authentication, you must configure the following parameter in ApsaraMQ for Kafka v2.x, but do not need to configure it for versions earlier than v2.x. 
   ssl.endpoint.identification.algorithm=

5. Configure the java.security.auth.login.config file.

   export KAFKA_OPTS="-Djava.security.auth.login.config=kafka_client_jaas.conf"                              

6. Run the following command to start the migration process:

   sh bin/kafka-mirror-maker.sh --consumer.config config/consumer.properties --producer.config config/producer.properties --whitelist topicName

Verify the result

You can check whether MirrorMaker runs properly by using one of the following methods:

• Run kafka-consumer-groups.sh to view the consumption progress of the user-created cluster.

  bin/kafka-consumer-groups.sh --new-consumer --describe --bootstrap-server endpoint of the user-created cluster --group test-consumer-group

• Send messages to the user-created cluster. In the ApsaraMQ for Kafka console, check the partition status of the topic, and check whether the total number of messages in the current broker is correct. You can view the specific content of a message in the ApsaraMQ for Kafka console. For more information, see Query messages.