Data Online Migration:Migration FAQ

How to estimate migration speed and duration

About migration speed:

• By default, Data Online Migration assigns each task a fixed theoretical peak bandwidth and QPS. For details, see Task bandwidth and QPS. Set the theoretical peak bandwidth and QPS based on your business needs to avoid competition between migration traffic and production traffic, which could affect normal operations.

• Migration speed is not guaranteed to reach the theoretical peak bandwidth and QPS. Key factors include the following:

  • Network link bandwidth and quality between the source and destination. Long-distance links with high latency or high packet loss rates (such as cross-border public network transfers) can reduce efficiency or cause frequent file transfer failures. Use a leased line or an acceleration link, such as an OSS acceleration endpoint, for better performance.

  • Read/write capabilities of the source and destination, and average file size. The migration service traverses directories, reads metadata and content from the source, and writes metadata and content to the destination while also reading metadata back. Source traversal capability affects scan speed. Read/write performance at either end affects migration speed. This is especially true when migrating millions of small files under 200 KB. Estimate average file size by dividing total source data volume by total file count.

  • Number of source directories and file distribution. For file systems that support directories (such as LocalFS or FTP), Data Online Migration scans directories concurrently level by level. Excessive directories or sparse file distribution slows down scanning.

  • Software and hardware configuration and number of proxy instances. If your source or destination data address uses a proxy, the proxy may become a bottleneck. For guidance on proxy selection, see Proxy selection.

• Due to these factors, Data Online Migration does not guarantee migration speed. Before running a full migration, validate your setup with a small-scale proof of concept (PoC) to accurately estimate real-world performance.

About migration duration:

• After estimating a reasonable migration speed, calculate migration time using total data volume and total file count. Estimate the fastest possible duration using total storage ÷ bandwidth and total file count ÷ QPS, then take the larger value.

• The calculated migration duration is theoretical. Account for bandwidth fluctuations, retries after failures, and other variables. Always prepare contingency plans and allow buffer time, especially for production migrations requiring downtime.

• Data Online Migration enforces upper limits on resource allocation per task. For details, see Task bandwidth and QPS.

The following is an example of a comprehensive evaluation:

• Scenario 1:

  • Question: The source is S3 with about 1 TB of data and 10 million files. The destination is OSS. The task is created in the Beijing region. How long will migration take?

  • Answer: According to Task bandwidth and QPS, the theoretical single-task speed in Beijing is 2 Gbps and 2,000 files/second. Calculations:

    • Duration by storage volume: 1 TB ÷ 2 Gbps ≈ 1.1 hours.

    • Duration by file count: 10,000,000 ÷ 2,000 files/second ≈ 1.4 hours.

    • Theoretical migration duration: Take the larger value: 1.4 hours.

• Scenario 2:

  • Question: The source is OSS with about 1 TB of data and 100,000 files. The destination is Alibaba Cloud NAS. The task is created in the Singapore region. How long will migration take?

  • Answer: According to Task bandwidth and QPS, the theoretical single-task speed in Singapore is 1 Gbps and 2,000 files/second. However, because the destination is LocalFS, the theoretical QPS limit is 200 files/second. Calculations:

    • Duration by storage volume: 1 TB ÷ 1 Gbps ≈ 2.3 hours.

    • Duration by file count: 100,000 ÷ 200 files/second ≈ 500 seconds.

    • Theoretical migration duration: Take the larger value: 2.3 hours.

Proxy shows abnormal status in the console

Data Online Migration proxy instances must be deployed to function properly. If the proxy status still shows Connection Abnormal after correct deployment, check the following:

• Network connectivity. Refer to the Network type parameter in Use a proxy for migration to verify connectivity from the proxy machine to Data Online Migration service domains.

• Incorrect credentials or insufficient permissions, such as:

  • Typographical errors. For example, an incomplete or corrupted AccessKeyId or SecretAccessKey due to copy-paste errors.

  • Insufficient permissions. The AccessKeyId must have one of the following permissions, and the Resource scope must be set to Account level:

    • mgw:VerifyAgentTunnel

    • AliyunOSSImportReadOnlyAccess

    • AliyunOSSImportFullAccess

  • Account mismatch. The AccessKeyId does not belong to the current RAM user or another RAM user under the same Alibaba Cloud account. Regardless of cross-account migration, you must use credentials from a RAM user under the current Alibaba Cloud account, and that RAM user must have one of the three permissions listed above.

• Credentials restricted by RAM Network access control policies. Data Online Migration requires that the AccessKeyId allows access from all network segments. To allow public network access, go to the AccessKey-level network access control policy page, select Enable, click Allow all public network access (the system automatically adds ::/0 and 0.0.0.0/0), click OK in the public network policy table, then click Submit at the bottom of the page.

• Credentials restricted by RAM access policies. If you assigned a custom policy to this RAM user that includes IP address restrictions, the proxy may fail to run.

• Inaccurate system clock on the proxy machine. The time must not deviate from real time by more than 5 minutes.

• Modified or incomplete deployment command. Copy and paste the auto-generated deployment command from the console exactly as shown. Do not modify it.

If the proxy status is unstable, possible causes include the following:

• Unstable network. Poor network quality between the proxy and Data Online Migration service causes intermittent connection issues. If the proxy’s network environment changes, recheck connectivity as described in Use a proxy for migration.

• High proxy load. A running migration task may cause occasional connection anomalies. If the task runs normally, this is expected. If the machine shows high load average, scale out proxy instances or lower the Maximum files migrated per second setting.

• Rate limiting enabled on the associated channel instance. Proxy health checks consume a small amount of channel QPS, which may compete with migration requests and get rejected. Increase the channel’s Requests per second limit or disable channel-level rate limiting.

• Proxy process has exited. See View, start, and stop proxy processes to check the process status. If the proxy machine’s instance type is below recommended specifications, the process may exit due to out of memory. For recommended specs, see Proxy selection.

• Duplicate deployment. Each proxy instance must run on only one machine. Deploying the same instance on multiple machines causes conflicts and instability. To switch machines, stop the old proxy process first, then redeploy.

Migration task shows no progress

• Background scheduling. Tasks typically wait 10–15 minutes in queue before starting. Wait patiently.

• Files filtered out. If the task uses a file filter and few source files match the criteria, progress updates slowly.

• Manifest pre-processing time. If the source is a manifest-type data address, pre-processing time increases with the total number of files in the manifest.

• Concurrent migration of large files. If the bandwidth graph on the task details page matches expected trends, this is normal.

• Other possible causes:

  • For LocalFS or FTP sources, too many directories or sparse file distribution slows scanning. Permission errors or directory corruption can also halt scanning.

  • For LocalFS destinations, migrating many gigabyte-sized files may delay progress updates. Check actual bandwidth on the proxy machine (for example, via ECS Cloud Monitor).

  • If either end uses a proxy, high load or connection issues on the proxy machine affect task progress.

How to view failed migration files

During migration, some files may fail due to insufficient source permissions, network issues, or other errors. Use one of these methods to identify failed files and their causes:

• Method 1: Push migration report

When creating a task, enable migration report delivery. You can also modify this setting later. After task completion, Data Online Migration writes a full migration record to a specified location in the destination data address. Review this report to identify failed files and reasons.

• Method 2: Push migration logs

When creating a migration task, enable migration log delivery. Data Online Migration pushes real-time logs to your Simple Log Service (SLS). Analyze these logs in SLS to find failed files and specific error details.

Common migration failure reasons

The following table lists error code keywords (partial) for migration failures. These codes may appear in:

• Error messages when creating a data address fails.

• Error messages in the task history when a task fails mid-run.

• The Failure reason field in migration reports.

underscoreUserMetaDropOrSkipUids=<UID>,drop;

Data volume mismatch after migration

If source and destination data volumes differ after migration completion, investigate the following.

If the destination data volume is smaller, possible causes include the following:

• Data changed during migration. The source may have received new writes, or the destination may have had deletions (manual or via lifecycle policies).

• Source contains multiple versions (such as OSS, S3, or NAS recycle bin). Data Online Migration only migrates the latest version.

• Statistics delay. Destination metrics may lag by hours or days. Consult the relevant documentation or use tools like ossutil for manual verification.

• Some data was skipped. For example, a 1 GB file on the source and a 100 MB file on the destination may be skipped based on the task’s overwrite settings.

• Source contains fragments. Data Online Migration does not migrate incomplete objects or fragments.

• Incorrect file filter configuration. If filters are enabled, verify rules for unintended spaces or special characters.

• Incorrect source Prefix. Specifying a Prefix migrates only data under that Prefix.

• Storage type differences. File systems like NAS count directories toward storage usage. Object storage like OSS has no directory concept, and empty objects do not count.

• Partial NAS mount failures. If the source is LocalFS with multiple proxies, and some NAS mounts are broken, files accessed via those proxies may be skipped due to NotFound errors, reducing total migrated data.

If the destination data volume is larger, possible causes include the following:

• Data changed during migration. The source may have had deletions (manual or via lifecycle policies), or the destination may have received new writes.

• Destination has multi-version feature enabled (such as OSS or NAS recycle bin). Data Online Migration uses non-multi-version APIs. If multi-version is enabled, each write creates a new version.

• Some data was skipped. For example, a 100 MB file on the source and a 1 GB file on the destination may be skipped based on overwrite settings.

• Residual fragments or temporary files on destination. If a task fails (due to user action, severe errors, or network issues), fragments or temp files may remain.

• Incorrect file filter configuration. Verify filter rules for unintended spaces or special characters.

• Existing data under destination Prefix. If the destination Prefix already contained data before migration, that data remains (though same-name files may be overwritten based on task settings).

• Storage type differences. File systems like NAS count directories toward storage usage. Object storage like OSS has no directory concept, and empty objects do not count.

• Migration report storage overhead. If you enabled migration report delivery, the report itself consumes space proportional to total migrated files.

Excess source egress traffic or repeated file migration

Normally, Data Online Migration downloads each source file once. If you observe higher-than-expected source egress traffic after migration, possible causes include the following:

• Internal retries. Network transient disconnections, read/write throttling, or proxy bottlenecks trigger limited automatic retries.

• External system access. Other systems (such as your application or CDN origin fetch) accessing the source during migration add to egress traffic.

If previously migrated files are re-migrated instead of skipped in subsequent runs, possible causes include the following:

• Incorrect Overwrite mode. If set to Force overwrite, Data Online Migration always overwrites destination files regardless of existing content.

• Destination lifecycle policies. If you selected Preserve last modified time during task creation, source file timestamps are applied to the destination. If the destination has a lifecycle policy that deletes files, migrated files may be removed shortly after.

• Frequent source updates. If overwrite mode is Overwrite based on last modified time, any timestamp update on the source (even without content change) triggers re-migration in the next run.

OSS cross-account migration issues

There are three cross-account scenarios: other account to your account, your account to other account, and other account to other account (complex and not recommended—convert to the first two). Authorization issues may arise. Summary:

• Core authorization steps:

  • Create a RAM role in your account via the RAM console. See Create a RAM role for data migration.

  • Share the RoleName with the administrator of the other account. The administrator grants permissions on the target Bucket. See Source Bucket authorization or Destination Bucket authorization.

  • Authorization is now complete. When creating the data address for the Bucket in Data Online Migration, enter the RoleName in the Authorization role field.

• After cross-account authorization, “insufficient permissions” errors may persist if the RoleName contains uppercase letters. Even if the original RoleName in RAM includes uppercase letters (for example, MyOssImportRole), use all lowercase letters (for example, myossimportrole) in the Bucket ACL policy.

NAS cross-account migration issues

• When using Data Online Migration, migrating NAS instances across different Alibaba Cloud accounts does not involve cross-account scenarios. The service uses standard POSIX I/O to read/write specified directories and does not recognize the underlying storage as NAS or its ownership.

• You must deploy a proxy and mount the NAS to the proxy, then treat it as a LocalFS data address. Pay close attention to directory mapping to avoid reading from or writing to unintended disk paths.

• Example: Migrate NAS from Account A to your account’s NAS.

  • Mount Account A’s NAS to machine M1. M1 can be an ECS instance under Account A (recommended) or any machine that supports mounting Account A’s NAS.

  • In the Data Online Migration console, create proxy instance agent1 and deploy it on M1.

  • Mount your account’s NAS to machine M2. M2 can be an ECS instance under your account (recommended) or any compatible machine.

  • In the Data Online Migration console, create proxy instance agent2 and deploy it on M2.

  • After confirming both proxies show “Normal” status, follow Migrate between LocalFS systems to create data addresses and migration tasks.

Does OSS support migration between Alibaba Cloud international and China accounts?

Yes. For details, see Migrate between Alibaba Cloud OSS instances.