This topic describes the architecture, migration tools, and procedures that are used to migrate a self-managed Kafka cluster to a Message Queue for Apache Kafka instance. This topic also describes the benefits of Message Queue for Apache Kafka and how migration works.

Benefits

For information about the benefits of migrating a self-managed Kafka cluster to Message Queue for Apache Kafka, see Benefits.

How migration works

When you migrate a self-managed Kafka cluster to Message Queue for Apache Kafka, you must use the original consumer to consume all messages that are queued in the original Apache Kafka cluster. Producers and consumers are deployed in a cluster. You can migrate the servers that are deployed in the cluster one by one. The migration process is transparent to your upper-layer services.

Migration methods

Description

  • After the migration process is complete, new topics and groups are created in the destination Message Queue for Apache Kafka instance. The configurations of the new topics and groups are the same as the configurations of the original topics and groups. The original topics and groups are retained in the self-managed Kafka cluster.
  • The topics and groups are migrated. The messages that are stored in the original topics, the subscription relationships of the original groups, and the consumer offset information of the original groups are not migrated.
  • When you migrate a self-managed Kafka cluster to a Message Queue for Apache Kafka instance, you can use the migration router feature in the Message Queue for Apache Kafka console or use migration tools. We recommend that you use the migration router feature to migrate a self-managed Kafka cluster to a Message Queue for Apache Kafka instance, and use a migration tool only when the public endpoint is unavailable for a self-managed Kafka cluster.

Architecture

The following figure shows how to migrate a self-managed Kafka cluster to a Message Queue for Apache Kafka instance.

Architecure

How to migrate the cluster:

  1. Obtain the public endpoint of the self-managed Apache cluster.
  2. Create a migration task in the Message Queue for Apache Kafka console.
  3. Create a consumer group in the Message Queue for Apache Kafka instance to consume messages that are sent to the instance.
  4. Create a producer in the Message Queue for Apache Kafka instance, and shut down the producer in the self-managed Kafka cluster. The system continues to use the original consumer group to consume the messages that are queued in the self-managed Kafka cluster.
  5. After all messages in the self-managed Kafka cluster are consumed by the original consumer group, shut down the consumer group and the self-managed Kafka cluster.

  1. Use a migration tool to migrate the data in the self-managed Kafka cluster to the Message Queue for Apache Kafka instance.
  2. Create a consumer group in the Message Queue for Apache Kafka instance to consume messages that are sent to the instance.
  3. Create a producer in the Message Queue for Apache Kafka instance, and shut down the producer in the self-managed Kafka cluster. The system continues to use the original consumer group to consume the messages that are queued in the self-managed Kafka cluster.
  4. After all messages in the self-managed Kafka cluster are consumed by the original consumer group, shut down the consumer group and the self-managed Kafka cluster.

Use the migration router feature

Message Queue for Apache Kafka provides a fully managed and O&M-free instance migration service for data synchronization between a self-managed instance and a Message Queue for Apache Kafka instance or between Message Queue for Apache Kafka instances. You can migrate instances in the Message Queue for Apache Kafka console without using migration tools. After the migration is complete, the metadata of the destination cluster is the same as the metadata of the source cluster. The metadata is continuously updated. For more information, see Migration router.

Use migration tools

Migration tools

Migration tool Description References
kafka-migration-assessment.jar.zip
  • You can use this migration tool to export the metadata of the self-managed Kafka cluster as a JSON-formatted file. In the Message Queue for Apache Kafka console, create a migration task and upload the JSON-formatted file. Then, run the task to migrate the metadata.
  • You can use this migration tool to migrate the topics and consumer groups of the self-managed Kafka cluster to the destination Message Queue for Apache Kafka instance.
MirrorMaker You can use this migration tool to replicate the image of the self-managed Kafka cluster to the destination Message Queue for Apache Kafka instance. Migrate data to the cloud

Procedure

MigrationProceduce
To migrate the self-managed Kafka cluster, perform the following steps:
  1. Optional: Evaluate instance specifications: If you know the specification information about the self-managed Kafka cluster that you want to migrate, such as the information about the traffic, disk capacity, disk type, and number of partitions of the cluster, you can determine the specification requirements of the destination Message Queue for Apache Kafka instance based on the information. For more information, see Evaluate instance specifications

    If you do not know the specification information about the self-managed Kafka cluster that you want to migrate, you can skip this step. When you migrate the metadata of the self-managed Kafka cluster to a new Message Queue for Apache Kafka instance, you can specify a migration tool to collect specification information about the self-managed Kafka cluster. Message Queue for Apache Kafka automatically evaluates the specification requirements of the destination instance based on the specification information that is collected by the migration tool.

  2. Migrate the self-managed Kafka cluster: You can run a command in your command-line interface (CLI) to migrate a specific type of resource in the self-managed Kafka cluster, or create a migration task to migrate all resources in the cluster.
    • Migrate the metadata of the self-managed Kafka cluster to a new Message Queue for Apache Kafka instance: In the Message Queue for Apache Kafka console, create and run a migration task. You need to purchase a Message Queue for Apache Kafka instance and use it as the destination instance. You can purchase an instance based on the specifications that are recommended by Message Queue for Apache Kafka. For more information, see Migrate the metadata of a self-managed Kafka cluster to a new Message Queue for Apache Kafka instance.
      Note If the metadata of the self-managed Kafka cluster contains the specification information about the traffic, disks, and other configurations of the cluster, Message Queue for Apache Kafka automatically evaluates and recommends specifications for the destination instance based on the metadata. You can purchase a Message Queue for Apache Kafka instance based on the instance specifications that are recommended by Message Queue for Apache Kafka.
    • Migrate the metadata of the self-managed Kafka cluster to an existing Message Queue for Apache Kafka instance: In the Message Queue for Apache Kafka console, create a migration task to migrate the metadata of the self-managed Kafka cluster to an existing Message Queue for Apache Kafka instance. For more information, see Migrate metadata.
    • Migrate a specific type of resource: If you want to migrate only the topics, groups, or messages in the self-managed Kafka cluster, you can use the corresponding migration method.
      1. Migrate the topics of the self-managed Kafka cluster to the destination Message Queue for Apache Kafka instance. For more information, see Migrate topics from a self-managed Kafka cluster to Message Queue for Apache Kafka.
      2. Migrate the consumer groups of the self-managed Apache Kafka cluster to the destination Group instance. For more information, see Migrate a group from a self-managed Kafka cluster to Message Queue for Apache Kafka.Message Queue for Apache Kafka
      3. Migrate the messages in the self-managed Kafka cluster to the destination Message Queue for Apache Kafka instance. This is optional. For more information, see Migrate data to the cloud.
        Important After messages in message queues are consumed, the messages are not reused. We recommend that you do not migrate messages unless you want to back up the messages in the self-managed Kafka cluster to the destination Message Queue for Apache Kafka instance.
  3. View the progress of the Message Queue for Apache Kafka migration task. For more information, see View the migration progress.
  4. View the resource list after the migration is complete. For more information, see Verify the migration result.