This topic describes how to create a migration task in the ApsaraMQ for Kafka console to migrate the metadata of a self-managed Kafka cluster to a ApsaraMQ for Kafka instance.
Prerequisites
- Java Development Kit (JDK) 8 is downloaded and installed. For more information, see Java Downloads.
- A ApsaraMQ for Kafka instance is purchased and deployed.
- For information about how to purchase an instance and deploy the instance in a virtual private cloud (VPC), see Purchase and deploy an instance that allows access from a VPC.
- For information about how to purchase an instance that can be accessed over the Internet and a VPC and deploy the instance, see Purchase and deploy an instance that allows access from the Internet and a VPC.
Background information
The metadata of an Apache Kafka cluster contains the configurations of topics and groups in the cluster. The metadata of an Apache Kafka cluster is stored in ZooKeeper. Each node of the Apache Kafka cluster obtains the most recent metadata of the cluster from ZooKeeper. This way, the metadata that is exported from each node of the cluster is the most recent data and is the same. You can export the metadata of the Apache Kafka cluster to a JSON file and then import the JSON file to another Apache Kafka cluster. This way, you can back up the metadata of the Apache Kafka cluster.
The metadata migration method is used to migrate the metadata of a self-managed Kafka cluster to an Alibaba Cloud ApsaraMQ for Kafka instance. You can export the metadata of a self-managed Kafka cluster and then import the metadata to a ApsaraMQ for Kafka instance. ApsaraMQ for Kafka automatically creates topics and groups in the destination ApsaraMQ for Kafka instance based on the metadata. This way, the metadata of the self-managed Kafka cluster is migrated to Message Queue for Apache Kafka.
Export metadata
Use the metadata export tool to export the metadata of the self-managed Kafka cluster.
- Click kafka-migration-assessment.jar to download the file that contains the metadata export tool.
- Upload the file that contains the metadata export tool to your self-managed Kafka cluster.
- Run the following commands in the directory in which the file is stored to extract the toolkit and allow the JAR file to be executed:
unzip -o kafka-migration-assessment.jar.zipchmod 777 kafka-migration-assessment.jar - Run the following command to export the metadata of the self-managed Kafka cluster:
java -jar kafka-migration-assessment.jar MigrationFromZk --sourceZkConnect <host:port> --sourceBootstrapServers <host:port> --targetDirectory ../xxx/ --fileName metadata.json --commitParameter Description Example sourceZkConnect The IP address and port of the ZooKeeper node on which the self-managed Kafka cluster is deployed. If you do not configure this parameter, the tool automatically obtains the IP address and the port number. 192.168.XX.XX:2181 sourceBootstrapServers The IP address and port of the self-managed Kafka cluster. If you do not configure this parameter, the tool automatically obtains the IP address and the port number. 192.168.XX.XX:9092 targetDirectory The directory in which you want to store the metadata file. If you do not configure this parameter, the metadata file is exported to the current directory by default. ../home/ fileName The name of the metadata file. If you do not configure this parameter, the default name kafka-metadata-export.json is used. metadata.json commit Commits the task. commit installTsar Specifies whether to install Taobao System Activity Reporter (Tsar). By default, Tsar is not automatically installed. Tsar can be used to obtain accurate information about the specifications of the machine that is used for the self-managed Kafka cluster and the recent memory usage, traffic, and configurations of the cluster. The automatic installation of Tsar requires a long period of time and may fail if the environment is incompatible.
N/A evaluate Specifies whether to obtain the specifications of the machine that is used for the self-managed Kafka cluster and the recent memory usage, traffic, and configurations of the cluster. ApsaraMQ for Kafka can evaluate the specification information and recommend specifications for the destination Message Queue for Apache Kafka instance based on the collected information. We recommend that you create a destination instance based on your business requirements The default value is true. This value specifies that the tool collects the specification information. If you do not want to use the specification evaluation feature, set the value of this parameter to false.
N/A After you export the metadata, the system generates a JSON file in the specified directory.
You can view the metadata file in the specified directory. You also can download and save the file to your on-premises machine.
Create a migration task
- Log on to the ApsaraMQ for Kafka console.
- In the Resource Distribution section of the Overview page, select the region where your instance is deployed.
- In the left-side navigation pane, click Migrate.
- On the Migration to Cloud page, click Create Task.
- In the Create Task section of the Create Task wizard, configure the parameters and click Next.
- Enter a task name in the Description field.
- Click Click here to upload a metadata file next to Metadata and upload the metadata file that you exported from the self-manage Apache Kafka cluster. The file must be in JSON format.
- In the Purchase Instance section of the Create Task wizard, click Next.
- In the Deploy Instance section of the Create Task wizard, select the destination ApsaraMQ for Kafka instance and click Create Task. On the Migrate page, you can view that the task is in the Migrating state. After the migration task is complete, you can use the topics and groups that Message Queue for Apache Kafka automatically created in the destination instance based on the migrated metadata.
Delete a migration task
On the Migrate page, find the migration task that you want to delete and click Delete in the Actions column.