When you move from a self-managed Apache Kafka cluster or another cloud provider to Alibaba Cloud, ApsaraMQ for Kafka provides a built-in cloud migration feature that synchronizes metadata (topics, consumer groups, ACLs, SASL users) and message data to your destination instance. The migration runs as a fully managed task -- no additional infrastructure or O&M is required. After a task starts, metadata on the destination instance stays in sync with the source and is continuously updated.
Prerequisites
Before you begin, ensure that you have:
An ApsaraMQ for Kafka instance in the Running state
Major version 2.2.0 or 2.6.2, with the latest minor version installed
One of the following instance editions:
Serverless instance
Professional Edition (High Write) or Professional Edition (High Read)
A source Apache Kafka cluster running version 1.0.0 or later
Check your instance status and version in the Basic Information section of the Instances page in the ApsaraMQ for Kafka console.
Limits
| Item | Limit |
|---|---|
| Migration tasks per instance | Up to 20. The combined total of cloud migration tasks and connector tasks cannot exceed 20. |
| Destination instance edition | Serverless instance, or Professional Edition (High Write) / Professional Edition (High Read) |
| Source Kafka version | Apache Kafka 1.0.0 or later |
Billing
The cloud migration feature is in public preview, independent of ApsaraMQ for Kafka instances, and free of charge. No service level agreement (SLA) is provided for this feature. For SLAs and billing of other Alibaba Cloud services required during migration, see the corresponding service documentation.
Set up network connectivity
Choose a network setup based on how your source Kafka cluster connects to the destination ApsaraMQ for Kafka instance.
VPC-to-VPC migration
If the source and destination instances are in different VPCs, connect the VPCs before you create a migration task.
China (Hangzhou) and China (Chengdu) regions: Automatic VPC connection is supported.
Other regions: Manually connect the VPCs. For details, see Connect VPCs in different regions.
Public network migration (self-managed cluster or cross-cloud instance)
If the source Kafka cluster is accessible only over the Internet, or the source and destination are in separate VPCs without direct connectivity, enable Internet access for the destination instance VPC.
Enable Internet access
Create a NAT gateway for the VPC where the ApsaraMQ for Kafka instance is deployed. For details, see Internet NAT gateway.
Bind an elastic IP address (EIP) to the NAT gateway. For details, see Associate an EIP with an ECS instance.
Create an SNAT entry for the vSwitch used by the ApsaraMQ for Kafka instance. For details, see Create and manage SNAT entries.
Create and deploy a cloud migration task
Step 1: Create the task
Log on to the ApsaraMQ for Kafka console. In the Resource Distribution section of the Overview page, select the region of the destination instance.
In the left-side navigation pane, click Migration. On the page that appears, click the Cloud Migration tab.
Click Create Task.
In the Create Cloud Migration Task panel, complete the following steps:
Configure Basic Information: Specify the Task Name and Destination Instance, then click Next.
Configure Source Service: Set the source connection parameters based on your source instance type, then click Next. For parameter descriptions, see Source service parameters below.
Configure Destination Service: Click Create.
Step 2: Deploy the task
On the Migration page, select the destination ApsaraMQ for Kafka instance from the Instance drop-down list.
Find the migration task and click Deploy in the Actions column.
When the Status column changes to Running, the migration task is active and synchronizing data.
Source service parameters
Configure the parameters based on your Source Instance Type selection.
VPC (Interconnection Between Self-managed Instance and VPC)
Choose this source type when the source Kafka cluster is in a VPC connected to the destination instance VPC.
| Parameter | Description | Example |
|---|---|---|
| Source Region | Region of the source instance. | China (Hangzhou) |
| Endpoint | Endpoint of the source Kafka cluster. | 192.168.XX.XX:9092 |
| VPC ID | VPC of the source instance. Auto-populated from the deployment configuration. | vpc-test-c**** |
| vSwitch ID | vSwitch used by the source instance. | vsw-bp1gbjqsf53og**** |
| Security Group | Security group of the source instance. | alikafka_pre-cn-7mz2**** |
| Security Protocol | Protocol used by the source cluster. See Security protocol options below. | PLAINTEXT |
| Number of Tasks | Number of synchronization threads. Valid values: 1, 6, 12. | 12 |
Public Network (IDC or Cross-cloud Instance)
Choose this source type when the source Kafka cluster is in an on-premises data center or another cloud provider, accessible over the Internet.
| Parameter | Description | Example |
|---|---|---|
| Endpoint | Endpoint of the source Kafka cluster. | 192.168.XX.XX:9092 |
| Security Group | Security group of the source instance. | alikafka_pre-cn-7mz2**** |
| Security Protocol | Protocol used by the source cluster. See Security protocol options below. | PLAINTEXT |
| Number of Tasks | Number of synchronization threads. Valid values: 1, 6, 12. | 12 |
Security protocol options
The Security Protocol parameter supports the following values:
PLAINTEXT -- No authentication or encryption.
SASL_PLAINTEXT -- SASL authentication without encryption. Requires:
SASL Username: The SASL username.
SASL Password: The SASL password.
Sasl_Mechanism: The SASL authentication mechanism. Valid values: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512.
SASL_SSL -- SASL authentication with SSL encryption. Requires:
SASL Username: The SASL username.
SASL Password: The SASL password.
Sasl_Mechanism: The SASL authentication mechanism. Valid values: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512.
SSL Truststore File: Upload the SSL certificate file.
SSL Truststore Password: Password for the SSL certificate.
SSL Endpoint Identification Algorithm: Algorithm used to verify the server certificate. Specify
https,http, or leave blank. Configuring this parameter helps prevent man-in-the-middle (MITM) attacks.
Runtime environment parameters
Click Configure Runtime Environment to access additional synchronization options. These parameters are available for both VPC and public network source types.
| Parameter | Description | Default |
|---|---|---|
| Synchronize SASL Users | Synchronize SASL user configurations from the source to the destination instance. | Yes |
| Synchronize Topic ACLs | Synchronize access control list (ACL) configurations attached to topics on the source instance. If the source is a self-managed cluster, create an SASL user on the destination instance first. | Yes |
| Synchronize Consumer Groups | Synchronize consumer groups from the source to the destination instance. | Yes |
| Synchronize Consumer Offsets | Synchronize consumer offsets from the source to the destination instance. Available only when Synchronize Consumer Groups is set to Yes. | Yes |
| Topic | Specific topics to synchronize. Leave blank to synchronize all topics. | All topics |
| Create Topics to Use Local Storage | Non-compacted topics to store using local storage on the destination instance. Leave blank to use cloud storage for all topics. | Cloud storage |
Manage migration tasks
On the Migration page, select the destination instance from the Instance drop-down list. Find the task and use the controls in the Actions column.
| Action | How to access | Description |
|---|---|---|
| View task details | Click Details. | View basic information, source service, destination service, and runtime environment configurations. |
| View synchronization progress | Click Synchronization Progress, select a topic, then click OK. | Monitor the sync status of individual topics. |
| Modify configuration | Click Modify Configuration. | Update task parameters. |
| Reset consumer offsets | Choose . Select a topic, specify the new offset, then click OK. | Reset offsets for specific topics. For details, see Reset consumer offsets. |
| Pause or resume | Choose or , then click OK. | Temporarily stop or restart synchronization. |
| Delete task | Choose , then click OK. | Permanently remove the migration task. |