Data migration between Milvus instances - Vector Retrieval Service for Milvus

You can use the data migration service to migrate data from a self-managed Milvus database to Vector Retrieval Service for Milvus. The self-managed database can be located in a local data center, on another cloud platform, or on an Alibaba Cloud ECS instance. This service also supports data migration between two Milvus instances. The service uses a full and incremental migration mechanism to ensure complete data sync. This topic describes how to securely and efficiently migrate data to Milvus using the migration between two Milvus instances as an example.

Restrictions and configuration requirements

Pre-migration preparations (required)

Operation status control

Cluster type	Requirement	Description
Source cluster	Stop all data modification operations	This includes write, delete, and update operations. Ensure the cluster is in a read-only state to prevent data inconsistencies during migration.
Destination cluster	Pause all data operations	This includes query, write, delete, and update operations. Keep the cluster unavailable to avoid data conflicts with the migration.

Version compatibility
Requirement
Specification
Source cluster version
Must be later than 2.3.6 (≥ v2.3.7)
Destination cluster version
Must be the same as or later than the source cluster version

Migration task limits

Task management
- Concurrency limit: Only one migration task can run at a time.
Data scope
- Database limit: Each migration task can migrate collections from only one database.
- Collection limit: Each migration task supports a maximum of five collections.
- Total data size: The total number of entities across all collections cannot exceed 500 million.
Data state
- Source instance requirement: The collections to be migrated must be in a loaded state.
- Destination instance requirement: The destination instance must be empty and contain no existing entity data.

Network requirements

If the two Milvus instances are in the same region, connect them using their private network addresses. This method does not require you to configure whitelists or security groups and provides faster data transmission.

If the two Milvus instances are in different regions, connect them using their public network addresses. You must add the public IP address range of the target instance's region to the public whitelist of the source instance.

Target instance region	IP address range (CIDR)
North China 2 (Beijing)	`39.105.32.0/24`
South China 1 (Shenzhen)	`120.77.174.0/24`
East China 2 (Shanghai)	`47.117.185.0/24`
China (Hong Kong)	`8.217.170.0/24`
China (Hangzhou)	`115.29.196.0/24`
Ulanqab	`121.89.83.0/24`
Singapore	`45.113.192.0/24`
Germany (Frankfurt)	`47.245.133.0/24`

Migration steps

Log on to the Alibaba Cloud Milvus console. Click Data Migration, and then click Milvus (Migrate via Endpoint).
Configure the connection information for the Source Cluster and select the target cluster. Then, click Next.
- Source Cluster Endpoint: http://<Milvus cluster internal address>.
- Username and Password: The username and password of the Milvus instance.
Select the source Database to migrate and add it to the target Database. Then, click Next.
Check and configure the field mapping. Then, click Confirm to start the migration task.

Monitoring and validation

You can view the migration progress and error logs in the console.
After the migration is complete, you can validate the data volume and retrieval consistency of the target collection.

Requirement	Specification
Source cluster version	Must be later than 2.3.6 (≥ v2.3.7)
Destination cluster version	Must be the same as or later than the source cluster version