Data Online Migration:Select a agent

How it works

Deploy an agent at the source

When you migrate data from self-managed storage, such as data in an IDC data center or third-party cloud storage, to Alibaba Cloud OSS, deploy an agent at the source.

• How it works: The agent acts as a secure entry point for reading data. Data Online Migration actively pulls your source data through this agent, forwards it to the Alibaba Cloud internal migration server, and then writes it to the destination OSS Bucket.

• Data stream: Data source → Agent ECS → Data Online Migration server → Destination OSS.

Deploy an agent at the destination

In some scenarios, such as when the destination is a local file system (LocalFs), you must deploy an agent at the destination.

• How it works: The agent acts as a secure exit point for writing data. Data Online Migration writes data to your local storage system through this agent.

• Data stream: OSS data source → Data Online Migration server → Agent ECS → Destination storage.

Step 1: Select a network solution

Select a network connectivity method between the agent ECS and Data Online Migration based on your migration scenario:

• VPC connection (Recommended): If the agent ECS and Data Online Migration are deployed in the same Alibaba Cloud region, we recommend using VPC for internal communication. This method provides higher security, greater stability, and lower latency. You must ensure that the agent ECS can access the internal same-region endpoint of Data Online Migration in that region. For example, if you use Data Online Migration in the China (Beijing) region, the agent must be able to access the internal same-region endpoint of Data Online Migration in the China (Beijing) region: {TunnelId}.cn-beijing.mgw-tc-internal.aliyuncs.com.

• Internet connection: If the agent ECS and Data Online Migration are in different regions, or if the agent is deployed on a local server or a third-party cloud, use the Internet for access. You must ensure that the agent has a public IP address and can access the public domain name of the service in that region. For example, if you use Data Online Migration in the China (Beijing) region, the agent must be able to access the public domain name of Data Online Migration in the China (Beijing) region: {TunnelId}.cn-beijing.mgw-tc.aliyuncs.com.

After you select a network solution, plan the instance type and quantity for the agent ECS.

Step 2: Plan the ECS selection

Select the appropriate CPU, memory, and network bandwidth configurations for the ECS based on the target migration bandwidth and data attributes:

1. CPU

• Minimum requirement: At least 4 cores.

• Recommended configuration: If you migrate a large number of small files (file processing queries per second (QPS) > 1,000 per second per agent), the CPU can become a bottleneck because of high-concurrency data migration. In this scenario, we recommend selecting a CPU with 8 or more cores.

2. Memory

• Minimum requirement: At least 8 GiB.

• Recommended configuration: Similarly, in a scenario with a large number of small files, high-concurrency data migration consumes a large amount of memory, which can lead to memory exhaustion. In this case, we recommend using at least 16 GiB of memory.

3. Network bandwidth

This is the most critical metric for determining the overall migration efficiency. You must calculate the required number of ECS instances based on the total bandwidth requirement.

• Core principle: Number of required ECS instances = Total target migration bandwidth / Network bandwidth of a single ECS

• Calculation example: Assume that the total target migration bandwidth is 100 Gbps and you selected the VPC solution in Step 1. In this case, you must focus on the internal bandwidth of the ECS.

  • If you select ecs.c7.2xlarge (internal bandwidth of 5 Gbps per instance), you need 100 / 5 = 20 instances.

  • If you select ecs.c7.4xlarge (internal bandwidth of 10 Gbps per instance), you need 100 / 10 = 10 instances.

For more information about ECS models, see Instance family. For recommended ECS instance types and their performance metrics, see Appendix: Recommended ECS instance types.

Appendix: Recommended ECS instance types