This topic describes the benefits of Message Queue for Apache Kafka instances and the working mechanism of migrating a self-managed Apache Kafka cluster to a ApsaraMQ for Kafka instance. This topic also describes the architectures, migration tools, and migration process.
Benefits
For information about the benefits of migrating a self-managed Apache Kafka cluster to ApsaraMQ for Kafka, see Benefits.
Working mechanism
When you migrate a self-managed Apache Kafka cluster to Message Queue for Apache Kafka, you must use the original consumer to consume all messages that are queued in the original Apache Kafka cluster. Producers and consumers are deployed in a cluster. You can migrate the servers that are deployed in the cluster one by one. The migration process is transparent to your upper-layer services.
Migration methods
Description
- After the migration process is complete, new topics and groups are created in the destination ApsaraMQ for Kafka instance. The configurations of the new topics and groups are the same as the configurations of the original topics and groups. The original topics and groups are retained in the self-managed Apache Kafka cluster.
- The topics and groups are migrated. The messages that are stored in the original topics, the subscription relationships of the original groups, and the consumer offset information of the original groups are not migrated.
- When you migrate a self-managed Apache Kafka cluster to a Message Queue for Apache Kafka instance, you can use the migration router feature in the Message Queue for Apache Kafka console or use a migration tool. We recommend that you use the migration router feature to migrate a self-managed Apache Kafka cluster to a Message Queue for Apache Kafka instance, and use a migration tool only when the public endpoint is unavailable for a self-managed Apache Kafka cluster.
Architectures
The following figure shows the architectures before and after you migrate a self-managed Apache Kafka cluster to a ApsaraMQ for Kafka instance.
Migration solutions:
Use the migration router feature
- Obtain the public endpoint of the self-managed Apache Kafka cluster.
- Create a migration task in the Message Queue for Apache Kafka console.
- Create a consumer group in the Message Queue for Apache Kafka instance to consume messages that are sent to the instance.
- Create a producer in the Message Queue for Apache Kafka instance, terminate the producer in the self-managed Apache Kafka cluster, and then continue to use the original consumer group to consume messages that are queued in the self-managed Apache Kafka cluster.
- After all messages in the self-managed Apache Kafka cluster are consumed by the original consumer group, terminate the consumer group and the self-managed Apache Kafka cluster.
Use a migration tool
- Use a migration tool to migrate the data of the self-managed Apache Kafka cluster to the Message Queue for Apache Kafka instance.
- Create a consumer group in the Message Queue for Apache Kafka instance to consume messages that are sent to the instance.
- Create a producer in the Message Queue for Apache Kafka instance, terminate the producer in the self-managed Apache Kafka cluster, and then continue to use the original consumer group to consume messages that are queued in the self-managed Apache Kafka cluster.
- After all messages in the self-managed Apache Kafka cluster are consumed by the original consumer group, terminate the consumer group and the self-managed Apache Kafka cluster.
Use the migration router feature
ApsaraMQ for Kafka provides a fully managed and O&M-free instance migration service for data synchronization between a self-managed instance and a Message Queue for Apache Kafka instance or between Message Queue for Apache Kafka instances. You can migrate instances in the Message Queue for Apache Kafka console without using a migration tool. After the migration is complete, the metadata of the destination cluster is the same as the metadata of the source cluster. The metadata is continuously updated. For more information, see Migration router.
Use a migration tool
Migration tools
| Migration tool | Description | References |
|---|---|---|
| kafka-migration-assessment.jar |
|
|
| MirrorMaker | You can use this migration tool to copy the image of the self-managed Apache Kafka cluster to the destination ApsaraMQ for Kafka instance. | Migrate data to the cloud |
Migration process
- Optional:Evaluate instance specifications: Evaluate instance specifications: If you know the specification information about the self-managed Apache Kafka cluster that you want to migrate, such as the traffic, disk capacity, disk type, and number of partitions of the cluster, you can determine the type of ApsaraMQ for Kafka instance that you need to purchase based on the information.
If you do not know the specification information about the self-managed Apache Kafka cluster that you want to migrate, you can skip this step. If you migrate the metadata of the self-managed Apache Kafka cluster to a new Message Queue for Apache Kafka instance, you can specify a migration tool to collect the specification information about the cluster. Message Queue for Apache Kafka automatically evaluates the specification requirements of the destination instance based on the information that is collected by the migration tool.
- Migrate the self-managed Apache Kafka cluster: You can run a command in your command-line interface (CLI) to migrate a specific type of resource in the self-managed Apache Kafka cluster, or create a migration task to migrate all resources in the cluster.
- Migrate the metadata of a self-managed Kafka cluster to a new Message Queue for Apache Kafka instance: In the ApsaraMQ for Kafka console, create a migration task, purchase a ApsaraMQ for Kafka instance based on the specifications that are recommended by the system, and then run the migration task. Note If the metadata of the self-managed Apache Kafka cluster that you want to migrate contains the specification information about the traffic, disks, and other configurations of the cluster, ApsaraMQ for Kafka automatically evaluates the specifications and recommends an instance type based on the metadata. You can purchase an instance based on the recommendation.
- Migrate the metadata of a self-managed Apache Kafka cluster to an existing Message Queue for Apache Kafka instance: In the ApsaraMQ for Kafka console, create a migration task to migrate the metadata of the self-managed Apache Kafka cluster to an existing ApsaraMQ for Kafka instance.
- Migrate a specific type of resource: If you want to migrate only topics, groups, or messages in the self-managed Apache Kafka cluster, you can use the corresponding migration method.
- For information about how to migrate topics in a self-managed Apache Kafka cluster to a ApsaraMQ for Kafka instance, see Migrate topics from a self-managed Kafka cluster to Message Queue for Apache Kafka.
- For information about how to migrate consumer groups in a self-managed Apache Kafka cluster to a ApsaraMQ for Kafka instance, see Migrate a group from a self-managed Kafka cluster to Message Queue for Apache Kafka.Group
- For information about how to migrate messages in a self-managed Apache Kafka cluster to a ApsaraMQ for Kafka instance, see Migrate data to the cloud. This is optional. Important After messages in message queues are consumed, the messages are not reused. We recommend that you do not migrate messages in a self-managed Apache Kafka cluster unless you want to back up the messages in the cluster to a ApsaraMQ for Kafka instance.
- Migrate the metadata of a self-managed Kafka cluster to a new Message Queue for Apache Kafka instance: In the ApsaraMQ for Kafka console, create a migration task, purchase a ApsaraMQ for Kafka instance based on the specifications that are recommended by the system, and then run the migration task.
- View the migration progress: View the progress of the migration task.
- Verify the migration result: View the resource list after the migration is complete.