Data Online Migration:Migrate data between OSS buckets

Before you begin

• Log on to the Data Online Migration console as a Resource Access Management (RAM) user created for data migration. For cross-account migration, log on as a RAM user created within the source or destination Alibaba Cloud account.

• Prepare RAM role authorization for both the source and destination buckets. For details, see Authorize a RAM role in the Data Online Migration console or Manual authorization.

• To push migration logs to Simple Log Service (SLS), activate SLS and complete the authorization on the Authorize page before you create a migration task.

Usage notes

• Source API behavior: Data Online Migration accesses the source data address through public APIs provided by the source storage service provider. Actual behavior depends on the provider's API implementation.

• Resource impact: Migration consumes resources at both the source and destination data addresses and may affect business workloads. Enable throttling or run tasks during off-peak hours to reduce impact.

• Overwrite behavior: Before a task starts, Data Online Migration compares files at the source and destination. If files at the source and destination have the same name, selecting the Overwrite All option overwrites the destination file. If same-name files contain different data and the destination file must be retained, rename one file or back up the destination file first.

• Lifecycle interaction: The source file's LastModifyTime attribute is retained after migration. If lifecycle rules are configured on the destination bucket, migrated files whose last modification time falls within a rule window may be deleted or archived to a specified storage class.

Limitations

• Static website hosting scan behavior: If static website hosting is enabled for source files, migration scans may detect directory objects that do not actually exist. For example, if myapp/resource/1.jpg exists, the scan may also return myapp/ and myapp/resource/. The non-existent directory objects fail to migrate, while myapp/resource/1.jpg migrates as expected.

• Symbolic links: Symbolic links at the source data address are migrated directly to the destination data address. For details, see Symbolic links.

• Task scope: Each task can migrate data from only one bucket. A single task cannot migrate all data in your account.

• Attribute migration support: Only specific attributes are migrated between OSS buckets.

  • Migrated: x-oss-meta-*, LastModifyTime, Content-Type, Cache-Control, Content-Encoding, Content-Disposition, Content-Language, and Expires.

  • Not migrated (including but not limited to): StorageClass, Acl, server-side encryption settings, Tagging, and user-defined x-oss-persistent-headers.

  Note

  This list is not exhaustive. Verify actual migration results to identify additional attributes that are not migrated.

Step 1: Select a region

The region you select in the Data Online Migration console determines whether you are charged for reading data from the source OSS bucket.

• If the source OSS bucket resides in the selected console region, no Internet read charges apply.

  Note

  For example, if the source and destination OSS buckets both reside in the China (Beijing) region and you select the China (Beijing) region in the Data Online Migration console, no fees are generated for reading data from the source OSS bucket over the Internet during migration.

  

• If the source OSS bucket resides in a different region from the selected console region, Internet read charges apply.

  Note

  For example, if you migrate data from an OSS bucket in the China (Beijing) region to an OSS bucket in the Singapore region, and you select the Singapore region in the Data Online Migration console, fees are generated for reading data from the source OSS bucket over the Internet.

  

Step 2: Create a source data address

1. In the left-side navigation pane, choose Data Online Migration > Address Management. On the Address Management page, click Create Address.

2. In the Create Address panel, configure the following parameters and click OK.

   Parameter	Required	Description
   Name	Yes	The name of the source data address. Must be 3 to 63 characters in length, case-sensitive. Can contain lowercase letters, digits, hyphens (-), and underscores (_). Encoded in UTF-8 format. Cannot start with a hyphen (-) or an underscore (_).
   Type	Yes	Select Alibaba OSS.
   Custom Domain Name	No	Specifies whether custom domain names are supported.
   Region	Yes	The region of the source data address. Example: China (Hangzhou).
   Authorize Role	Yes	Grant permissions to a RAM role for accessing the source bucket. The method depends on whether the source bucket belongs to the current account. Same account: Create and authorize a RAM role in the Data Online Migration console. For details, see Authorize a RAM role in the Data Online Migration console. Alternatively, manually attach policies in the RAM console. For details, see Manual authorization. Cross-account: Attach policies to a RAM role in the OSS console. For details, see External source bucket.
   Bucket	Yes	The name of the OSS bucket that stores the data to migrate.
   Prefix	No	The prefix of the source data address. Specify a prefix to migrate specific data. Cannot start with a forward slash (/) but must end with one (/). Example: data/to/oss/. With prefix: If you set the source prefix to example/src/ and the destination prefix to example/dest/, a file named example.jpg in example/src/ is migrated to example/dest/example.jpg. Without prefix: If no source prefix is specified, the full source path is preserved. For example, srcbucket/example.jpg with a destination prefix of destbucket/ is migrated to destbucket/srcbucket/example.jpg.
   Tunnel	No	The tunnel to use. Important Required only when you migrate data by using Express Connect circuits or VPN gateways, or migrate data from self-managed databases to the cloud. If data at the destination data address is stored in a local file system, or you need to migrate data over an Express Connect circuit in an environment such as Alibaba Finance Cloud or Apsara Stack, create and deploy an agent.
   Agent	No	The agent to use. Important Required only when you migrate data by using Express Connect circuits or VPN gateways, or migrate data from self-managed databases to the cloud. Select up to 200 agents for a specific tunnel at a time.

Step 3: Create a destination data address

1. In the left-side navigation pane, choose Data Online Migration > Address Management. On the Address Management page, click Create Address.

2. In the Create Address panel, configure the following parameters and click OK.

   Parameter	Required	Description
   Name	Yes	The name of the destination data address. Must be 3 to 63 characters in length, case-sensitive. Can contain lowercase letters, digits, hyphens (-), and underscores (_). Encoded in UTF-8 format. Cannot start with a hyphen (-) or an underscore (_).
   Type	Yes	Select Alibaba OSS.
   Custom Domain Name	No	Specifies whether custom domain names are supported.
   Region	Yes	The region of the destination data address. Example: China (Hangzhou).
   Authorize Role	Yes	Grant permissions to a RAM role for accessing the destination bucket. The method depends on whether the destination bucket belongs to the current account. Same account: Create and authorize a RAM role in the Data Online Migration console. For details, see Authorize a RAM role in the Data Online Migration console. Alternatively, manually attach policies in the RAM console. For details, see "Step 4: Grant permissions on the destination bucket to the RAM role" in the Preparations topic. Cross-account: Attach policies to a RAM role in the OSS console. For details, see "Step 4: Grant permissions on the destination bucket to the RAM role" in Preparations topic.
   Bucket	Yes	The name of the OSS bucket to which the data is migrated.
   Prefix	No	The prefix of the destination data address. Specify a prefix to migrate data to a specific path. Cannot start with a forward slash (/) but must end with one (/). Example: data/to/oss/. With prefix: If you set the source prefix to example/src/ and the destination prefix to example/dest/, a file named example.jpg in example/src/ is migrated to example/dest/example.jpg. Without prefix: If no destination prefix is specified, the source data is migrated to the root directory of the destination bucket.
   Tunnel	No	The tunnel to use. Important Required only when you migrate data by using Express Connect circuits or VPN gateways, or migrate data from self-managed databases to the cloud. If data at the destination data address is stored in a local file system, or you need to migrate data over an Express Connect circuit in an environment such as Alibaba Finance Cloud or Apsara Stack, create and deploy an agent.
   Agent	No	The agent to use. Important Required only when you migrate data by using Express Connect circuits or VPN gateways, or migrate data from self-managed databases to the cloud. Select up to 200 agents for a specific tunnel at a time.

Step 4: Create a migration task

1. In the left-side navigation pane, choose Data Online Migration > Migration Tasks. On the Migration Tasks page, click Create Task.

2. In the Select Address step, configure the following parameters.

   Parameter	Required	Description
   Name	Yes	The name of the migration task. Must be 3 to 63 characters in length, case-sensitive. Can contain lowercase letters, digits, hyphens (-), and underscores (_). Encoded in UTF-8 format. Cannot start with a hyphen (-) or an underscore (_).
   Source Address	Yes	The source data address that you created.
   Destination Address	Yes	The destination data address that you created.

3. In the Task Configurations step, configure the following parameters.

   Parameter	Required	Description
   Basic Settings
   Migration Bandwidth	No	The maximum bandwidth for the migration task. Options: Default: Use the default upper limit. Actual bandwidth depends on file size and file count. Specify an upper limit: Set a custom upper limit as prompted. Important Actual migration speed depends on the source data address, network conditions, destination throttling, and file size. The actual speed may not reach the specified limit. Set a reasonable value based on your source data address, migration purpose, business needs, and network bandwidth. Inappropriate throttling may affect business performance.
   Files Migrated Per Second	No	The maximum number of files migrated per second. Options: Default: Use the default upper limit. Specify an upper limit: Set a custom upper limit as prompted. Important Actual migration speed depends on the source data address, network conditions, destination throttling, and file size. The actual speed may not reach the specified limit. Set a reasonable value based on your source data address, migration purpose, business needs, and network bandwidth. Inappropriate throttling may affect business performance.
   Overwrite Mode	Yes	Whether to overwrite a destination file with the same name as a source file. Options: Do not overwrite: Does not migrate the source file. Overwrite All: Overwrites the destination file. Overwrite based on the last modification time: Overwrites the destination file if the source file was modified more recently. If the modification times match, overwrites the destination file when the files differ in size or Content-Type header. Warning If you select Overwrite based on the last modification time, newer files may still be overwritten by older ones, which risks losing recent updates. Make sure source files contain the last modification time, size, and Content-Type header. Otherwise, the overwrite policy may produce unexpected results. If you select Do not overwrite or Overwrite based on the last modification time, the system retrieves meta information from the source and destination to decide whether to overwrite. Request fees apply for both data addresses.
   Audit Mode
   Migration Report	Yes	Whether to push a migration report. Options: Do not push (default): Does not push the report to the destination bucket. Push: Pushes the report to the destination bucket. For details, see What to do next. Important The report occupies storage space at the destination. It may be pushed with a delay. Wait until the report is generated. A unique ID is generated for each execution. A report is pushed only once. Do not delete the report unless necessary.
   Migration Logs	Yes	Whether to push migration logs to SLS. Options: Do not push (default): Does not push logs. Push: Pushes logs to SLS. View logs in the SLS console. Push only file error logs: Pushes only error logs to SLS. View error logs in the SLS console. If you select Push or Push only file error logs, Data Online Migration creates an SLS project named aliyun-oss-import-log-{Alibaba Cloud account ID}-{Region}. Example: aliyun-oss-import-log-137918634953****-cn-hangzhou. Important Before selecting Push or Push only file error logs, activate SLS and confirm the authorization on the Authorize page.
   Authorize	No	Displayed when Migration Logs is set to Push or Push only file error logs. Click Authorize to go to the Cloud Resource Access Authorization page. Click Confirm Authorization Policy. The RAM role AliyunOSSImportSlsAuditRole is created and permissions are granted.
   Filter
   File Name	No	Filter files by name. Supports both inclusion and exclusion rules. Only re2 regular expression syntax is supported. For details, visit re2. Examples: .*\.jpg$: All files ending with .jpg. ^file.*: All files starting with file in the root directory. If a prefix is configured for the source data address (for example, data/to/oss/), use ^data/to/oss/file.* to match files starting with file in that directory. .*/picture/.* : Files whose paths contain a subdirectory called picture. Important If an inclusion rule is configured, all matching files are migrated. Multiple inclusion rules are evaluated with OR logic. If an exclusion rule is configured, all matching files are skipped. Multiple exclusion rules are evaluated with OR logic. Exclusion rules take precedence over inclusion rules. If a file matches both, the file is not migrated.
   File Modification Time	No	Filter files by last modification time. If a time range is specified, only files whose last modified time falls within that range will be migrated, as follows: If only the start time is set to Jan 1, 2019 and no end time is specified, only files with a last modified time on or after Jan 1, 2019 will be migrated. If only the end time is set to Jan 1, 2022 and no start time is specified, only files with a last modified time on or before Jan 1, 2022 will be migrated. If the start time is Jan 1, 2019 and the end time is Jan 1, 2022, only files with a last modified time between Jan 1, 2019 and Jan 1, 2022 (inclusive) will be migrated.
   Migration Configurations
   Last Modified Time	Yes	Whether to retain the source file’s last modified time. Yes (default): The source file’s last modified time is preserved on the destination file. No: The last modified time is not preserved.
   Convert from Appendable to Normal or Multipart	No	Whether to convert Appendable objects to Normal or Multipart objects. Yes: Appendable objects are converted to Normal or Multipart when migrated to the destination. No: Objects are not converted. Appendable objects remain Appendable after migration. Important This option is available only when whitelist access is enabled. Currently, only migrations from OSS to OSS support this option.
   Add Target Storage Class	No	Whether to specify the storage class for destination files. Specified: Migrated files are set to the specified storage class. Available classes: Standard, Infrequent Access, Archive, Cold Archive, Deep Cold Archive. Not specified (default): No change is made. Files use the destination’s default storage class. Important This option is available only when whitelist access is enabled. Currently, only tasks with OSS as the destination support this option.
   Task Scheduling
   Execution Time	No	The schedule for running the migration task. Options: Immediately: The task runs right away. Scheduled Task: The task runs within a specified time period every day. The task starts at the specified start time and stops at the specified stop time. Periodic Scheduling: The task runs based on the execution frequency and number of executions you specify. Execution Frequency: Every Hour, Every Day, Every Week, Certain Days of the Week, or Custom. For details, see Supported execution frequencies. Executions: The maximum number of times the task runs. If not specified, the task runs once. Important If the current execution is not complete by the next scheduled start time, the task starts its next execution at the subsequent scheduled start time after the current migration completes. This continues until the task runs the specified number of times. Concurrent task limits by region: up to 10 in China, up to 5 outside China. Exceeding these limits may delay tasks or prevent them from completing on schedule. You can manually start and stop tasks at any time, regardless of the execution schedule.

4. Read and confirm the Data Online Migration agreement. Then click Next.

5. Verify the configurations and click OK. The migration task is created.

Step 5: Verify data