Migrate self-managed Apache Kafka clusters to ApsaraMQ for Kafka instances - ApsaraMQ for Kafka

If you want to reduce costs, increase scalability, and improve reliability for your self-managed Apache Kafka cluster, you can migrate the cluster to an ApsaraMQ for Kafka instance. ApsaraMQ for Kafka is a fully managed and maintenance-free service that allows you to focus on business development without the need for deployment and maintenance.

Usage notes

After the migration is complete, the source topics and groups in the self-managed Apache Kafka cluster are not deleted. Instead, new topics and groups with the same configurations are created in the destination ApsaraMQ for Kafka instance.
You can choose whether to migrate message data and offset information based on your business requirements.

Solution description

When you migrate a self-managed Apache Kafka cluster to an ApsaraMQ for Kafka instance, you must use the original consumers to consume all messages that are queued in the original cluster. Producers and consumers are deployed in clusters. You can migrate servers that are deployed in the clusters one by one. The migration process does not affect your upper-layer services. For information about the benefits of migrating a self-managed Apache Kafka cluster to an ApsaraMQ for Kafka instance, see Benefits.

The following figure shows how to migrate a self-managed Apache Kafka cluster to an ApsaraMQ for Kafka instance.

ApsaraMQ for Kafka provides the following solutions:

Item	Cloud migration	Metadata import	Tool-assisted migration
Migration method	Data is migrated by creating a migration task in the ApsaraMQ for Kafka console. In most cases, we recommend that you use this method to migrate data.	Data is exported as a JSON file using the `kafka-migration-assessment.jar` tool and then migrated by creating a metadata import task in the ApsaraMQ for Kafka console. Note If your self-managed Apache Kafka cluster does not support Internet access, you can use this method to migrate data.	Data is migrated using the `kafka-migration-assessment.jar` or `MirrorMaker` tool.
Migration content	Topics Groups Message data and offset information Access control lists (ACLs) attached to topics Before migration, you must create Simple Authentication and Security Layer (SASL) users in the ApsaraMQ for Kafka console.	Topics Groups	`kafka-migration-assessment.jar` Topics Groups `MirrorMaker` Message data and offset information
Migration process	Obtain the public endpoint of the self-managed Apache Kafka cluster. Create a cloud migration task in the ApsaraMQ for Kafka console. Start consumer groups on the ApsaraMQ for Kafka instance to consume messages that are sent to the instance. Start producers on the ApsaraMQ for Kafka instance, terminate the producers in the self-managed Apache Kafka cluster, and then use the original consumer groups to consume messages that are queued in the self-managed Apache Kafka cluster. After all messages in the self-managed Apache Kafka cluster are consumed by the original consumer groups, terminate the consumer groups and the self-managed Apache Kafka cluster.	Use a migration tool to migrate data in the self-managed Apache Kafka cluster to the ApsaraMQ for Kafka instance. Start consumer groups on the ApsaraMQ for Kafka instance to consume messages that are sent to the instance. Start producers on the ApsaraMQ for Kafka instance, terminate the producers in the self-managed Apache Kafka cluster, and then use the original consumer groups to consume messages that are queued in the self-managed Apache Kafka cluster. After all messages in the self-managed Apache Kafka cluster are consumed by the original consumer groups, terminate the consumer groups and the self-managed Apache Kafka cluster.

ApsaraMQ for Kafka:Migrate self-managed Apache Kafka clusters to ApsaraMQ for Kafka instances

Usage notes

Solution description

References