You can use the Alibaba Cloud Milvus data migration service to migrate data from self-managed Milvus databases in an on-premises data center, on another cloud platform, or on Alibaba Cloud ECS instances to Alibaba Cloud Milvus. The service also supports data migration between two Alibaba Cloud Milvus instances, including instances in the same account or in different accounts. It uses full migration to ensure that all data is synchronized. This topic uses migration between Alibaba Cloud Milvus instances as an example to describe how to migrate data securely and efficiently.
Limitations and requirements
Pre-migration preparations (required)
-
Operational state control
Cluster type
Requirement
Description
Source cluster
Stop all data modification operations.
Halt all write, delete, and update operations to ensure the cluster is in a read-only state. This prevents data inconsistencies during migration.
Target cluster
Pause all data operations.
Halt all query, write, delete, and update operations. Keep the instance unavailable to prevent conflicts with incoming data.
-
Version compatibility
Requirement
Specification
Source cluster version
Must be later than 2.3.6 (that is, ≥ v2.3.7)
Target cluster version
Must not be earlier than the source cluster version
Migration task limits
-
Task management
-
Concurrency limit: Only one migration task can run at a time.
-
-
Data scope
-
Database limit: Each migration task can migrate collections from only one database.
-
Collection limit: Each migration task supports a maximum of five collections.
-
Total data size: The total number of entities across all collections cannot exceed 500 million.
-
-
Data state
-
Source instance requirement: The collections to be migrated must be in the loaded state.
-
Destination instance requirement: The destination instance must be empty with no existing entity data.
-
Network requirements
-
The two Milvus instances are in the same region: Connect to the instance over the internal address. You do not need to configure an allowlist or security group, and data transfer is faster.
-
The two Milvus instances are in different regions: Connect to the instance over the public address. You must add the public IP address range of the region where the target instance is located to the public access whitelist of the source instance.
Region of the target instance
IP address range (CIDR)
China (Beijing)
39.105.32.0/24China (Shenzhen)
120.77.174.0/24China (Shanghai)
47.117.185.0/24China (Hong Kong)
8.217.170.0/24China (Hangzhou)
115.29.196.0/24China (Ulanqab)
121.89.83.0/24Singapore (Asia Pacific Southeast)
8.222.220.0/24Germany (Frankfurt)
47.245.133.0/24 -
The two Milvus instances are under different accounts (cross-account migration): The EndPoint field of the source cluster supports the connection address of a Milvus instance under another Alibaba Cloud account. For cross-account migration, make sure that the target instance can access the EndPoint address of the source instance over the network. If you use a public address for the connection, add the public IP address range of the region where the target instance is located to the public access whitelist of the source instance.
Migration steps
-
Log on to the Alibaba Cloud Milvus console, click Data Migration, and then click Milvus (via Endpoint).
-
Configure the connection information for the Source Cluster and select the Target Cluster, and then click Next.
-
Source Cluster Endpoint: The connection address of the source Milvus cluster. For same-account and same-region scenarios, enter the internal address. For cross-account or cross-region scenarios, enter the public address or an internal address with a connected network.
On the Instance Details page of the source instance, go to the Access Address section to obtain the Internal Address.
-
Username and Password: The username and password of the source Milvus instance. For cross-account migration, enter the authentication information of the Milvus instance under the source account.
The following notes are displayed at the top of the page:
-
The source cluster version must be later than 2.3.6.
-
Complete the cluster network configuration before migration.
-
Migration from a higher version to a lower version is not supported.
In the Target Cluster section, select the target Milvus instance from the drop-down list.
-
-
-
Select the Database to migrate and add it to the target Database, and then click Next.
When you configure the data migration, make sure that the collections to migrate in the source cluster are in the load state.
-
Check and configure the field mapping, and then click OK to start the migration task.
Monitoring and verification
-
View the migration progress and error logs in the console
In the task list on the Data Migration page, you can view the status and progress of the migration task. When the status column shows Success and the progress bar is full, the data migration is complete.
-
After the migration is complete, we recommend that you verify the data volume and recall consistency of the target collection.