You can use the data migration service to migrate data from a self-managed Milvus database to Vector Retrieval Service for Milvus. The source database can be located in an on-premises data center, on another cloud platform, or on an Alibaba Cloud ECS instance. The service also supports data migration between different Vector Retrieval Service for Milvus instances. It uses a full and incremental migration mechanism to ensure complete data synchronization. This guide demonstrates how to securely and efficiently migrate your data to Vector Retrieval Service for Milvus from a self-managed Milvus on an ECS instance.
Limitations and requirements
Pre-migration preparations (required)
-
Operation status control
Cluster type
Requirement
Description
Source cluster
Stop all data modification operations
Includes write, delete, and update operations. The cluster must be in a read-only state to prevent data inconsistencies during migration.
Destination cluster
Pause all data operations
Includes query, write, delete, and update operations. The cluster must remain unavailable to avoid data conflicts during migration.
-
Version compatibility
Requirement
Specification
Source cluster version
Must be later than 2.3.6 (≥ v2.3.7)
Destination cluster version
Must be the same as or later than the source cluster version
Migration task limits
-
Task management
-
Concurrency limit: Only one migration task can run at a time.
-
-
Data scope
-
Database limit: Each migration task can migrate collections from only one database.
-
Collection limit: Each migration task supports a maximum of five collections.
-
Total data size: The total number of entities across all collections cannot exceed 500 million.
-
-
Data state
-
Source instance requirement: The collections to be migrated must be in the loaded state.
-
Destination instance requirement: The destination instance must be empty with no existing entity data.
-
Vector Retrieval Service for Milvus 2.6 is backward compatible with version 2.5. If your source cluster runs a 2.5.x version, such as 2.5.12, you can migrate its data directly to a 2.6 destination instance without converting the data format.
Network security configuration
The data migration service reads data from your source cluster and transfers it to the destination cluster by using a fixed set of IP addresses. To ensure a successful migration, your source cluster must allow inbound connections from these IP address ranges (CIDRs). Based on the region of your destination instance, configure your database firewall, security group, or access control list (ACL) to accept traffic from the corresponding IP range. For example, if your destination instance is in the China (Hangzhou) region, add the IP address range for China (Hangzhou) to your ECS security group.
|
Region |
IP address range (CIDR) |
|
China (Beijing) |
|
|
China (Shenzhen) |
|
|
China (Shanghai) |
|
|
China (Hong Kong) |
|
|
China (Hangzhou) |
|
|
China (Ulanqab) |
|
|
Singapore |
|
|
Germany (Frankfurt) |
|
Migration procedure
Step 1: Configure the firewall allowlist
In the security group of the server where the source Milvus service is deployed, open the default port 19530 and configure an inbound rule that allows access only from the fixed IP address range (CIDR) of the data migration service.
The following example shows how to configure the security group rule for a self-managed Milvus on an ECS instance:
Set Authorization Policy to Allow, set Priority to 1, set Protocol to Custom TCP, enter the CIDR of the data migration service 115.29.196.0/24 in the Source field, enter 19530 for the Destination (this instance) port, and then click Submit.
Step 2: Start the data migration task
-
Log on to the Vector Retrieval Service for Milvus console, click Data Migration, and then click Milvus (via Endpoint).
-
Configure the Source Cluster connection information, select a Target Cluster, and then click Next.
If the source cluster is a self-managed Milvus on an ECS instance:
-
Source Cluster Endpoint: http://ECS_Public_IP:9091
-
Username and Password: The username and password of the self-managed Milvus on the ECS instance.
The prompt at the top of the configuration page indicates: the source cluster version must be 2.3.6 or later, network configuration must be completed in advance, you can use the VTS tool if the network is not reachable, and migration from a later version to an earlier version is not supported. After you complete the configuration, click Next.
When you use a third-party migration tool such as Milvus Migration Tool to connect to a Vector Retrieval Service for Milvus instance, the Token parameter must be in the format of
username:password(for example,root:YourPassword123). The default administrator username is root, which is specified when the instance is created and cannot be changed. The password is the logon password that is set when the instance is created. To reset the password, go to the Details page, click the Security Configuration tab, and click Reset Root Password in the cluster configuration section. -
-
Select the Database to migrate and add it to the target Database, and then click Next.
Make sure that the collections to be migrated in the source cluster are in the loaded state. In the Migration Source panel, select the database and collection, and use the arrow buttons to add them to the Migration Target panel on the right.
-
Review and configure the field mapping, and then click OK to start the migration task.
On the field mapping page, both the source collection and the target collection are minimal_vectors. The fields include id (Int64, primary key, auto ID) and vector (FloatVector). The Dynamic Column toggle is turned on, and the Shard value is 2.
Click the gear icon next to a field in the target collection to configure advanced options in the pop-up panel, including Auto ID (automatic ID generation toggle), Default Value, and Partition Key. The Partition Key supports the INT64 or VARCHAR type. You can select at most one field as the partition key, and the field cannot be set to support null values.
Step 3: Monitor and verify
-
View the migration progress and error logs on the console.
After the migration task is completed, the Status column of the task displays Succeeded and the Progress column shows a full green progress bar on the Data Migration page, indicating that the migration is successfully completed.
-
After the migration is completed, we recommend that you verify the data volume and recall consistency of the target collection.