All Products
Search

This Product

    Migration Hub:Migrate a self-managed Kafka cluster to a self-managed Kafka cluster on Alibaba Cloud

    Document Center

    Migration Hub:Migrate a self-managed Kafka cluster to a self-managed Kafka cluster on Alibaba Cloud

    Last Updated:Nov 24, 2025

    1. Scenarios

    This solution applies to migrating a self-managed Kafka cluster from another cloud provider or an Internet Data Center (IDC) to a self-managed Kafka cluster on Alibaba Cloud.

    The migration type is metadata migration.

    2. Migration tools

    1. MirrorMaker

    MirrorMaker is a tool provided by Apache Kafka for data replication across clusters. It comes in two versions:

    • MirrorMaker 1: Implemented using the kafka-mirror-maker.sh script. It supports only message migration and does not support metadata synchronization.

    • MirrorMaker 2: Implemented using the connect-mirror-maker.sh script. It supports both metadata and message migration and depends on the Kafka Connect framework.

    To determine which MirrorMaker version to use, check your source Kafka version. If the connect-mirror-maker.sh file exists in the kafka/bin/ folder, you can use MirrorMaker 2. Otherwise, use MirrorMaker 1.

    3. Migration plan

    1. Migration flow

    • Instance creation: Deploy a self-managed Kafka cluster on an Alibaba Cloud ECS instance and complete the basic environment configuration.

    • Data migration:

      • Metadata migration: Use MirrorMaker 2 to synchronize metadata, such as topic configurations, number of partitions, number of replicas, and Access Control List (ACL) permissions.

    • Data validation: Verify the consistency of the number of topics, number of partitions, number of messages, and message content.

    2. Data migration plan

    2.1. Use MirrorMaker

    Migration principle:

    Use the consumer and producer modules of MirrorMaker to pull messages from the source Kafka cluster and write them to the destination Kafka cluster. This process achieves data synchronization.

    Prerequisites:

    • The source Kafka cluster must be accessible, and you must open the required network ports.

    • The destination Kafka cluster must be deployed and running.

    • Ensure that the MirrorMaker version is compatible with the source Kafka version.

    Risks and notes:

    • During data migration, you must monitor network bandwidth, disk I/O, and the Kafka cluster payload.

    • If the source Kafka cluster has uncommitted transactional messages, you must ensure transactional consistency.

    Advantages:

    • It supports full and incremental migration to meet the needs of large-scale data synchronization.

    • It provides flexible control over the migration scope through configuration, such as specifying a topic whitelist.

    3. Data validation plan

    3.1. Metadata validation

    • You can use the kafka-topics.sh --describe command to check whether the topic configurations on the destination cluster are consistent with the source cluster.

    • You can compare metadata information, such as the number of partitions, number of replicas, and ACL permissions.

    4. Migration procedure

    1. Migrating metadata and messages using the migration tool in the Alibaba Cloud RocketMQ console

    The migration is performed using MirrorMaker. The tool is `kafka-mirror-maker.sh`, which is located in the `bin` folder.

    Step 1: Configure the consumer and producer

    The key to the migration is to configure the MirrorMaker consumer and producer. The configuration files are `consumer.properties` and `producer.properties`, which are located in the `config` folder.

    The `consumer.properties` file is configured as follows:

    # Note: Configure the address of the source cluster here.
bootstrap.servers=172.20.77.74:9092 

# consumer group id
group.id=migration-group

auto.offset.reset=earliest

    The `producer.properties` file is configured as follows:

    # format: host1:port1,host2:port2 ...
# Note: Configure the address of the destination cluster here.
bootstrap.servers=172.20.77.91:9092

    Step 2: Execute the metadata migration

    After the configuration is complete, you can run the migration command. In the command:

    --whitelist: Specifies a comma-separated list of topics to migrate.

    ./kafka-mirror-maker.sh --consumer.config ../config/consumer.properties --producer.config ../config/producer.properties --whitelist="migration-to-aliyun"

    Step 3: Monitor the migration status

    • You can check the MirrorMaker log (logs/mirror-maker.log) to confirm that there are no faults.

    • You can use the kafka-topics.sh --list command to verify that the topics are created successfully on the destination cluster.

    5. Data validation procedure

    1. Metadata validation

    You can use the `kafka-topics.sh --describe --zookeeper <zk_host>` command to compare the topic configurations of the source and destination clusters. You can also use the `kafka-acls.sh` command to verify that ACL permissions are synchronized.

    6. Migration notes

    1. Version compatibility: Ensure that the source and destination Kafka versions are compatible to prevent migration failures caused by protocol differences.

    2. Network connectivity: Maintain a stable network connection between the source and destination clusters during the migration to avoid interruptions.

    3. Data consistency: Avoid changes to the source data during the migration. Alternatively, you can use incremental synchronization to ensure consistency.

    4. Resource reservation: Reserve sufficient CPU, memory, and bandwidth resources during the migration to prevent performance bottlenecks.

    5. Rollback plan: Create a rollback plan to handle migration failures. For example, you can retain the source data to quickly recover your services.