:Migration implementation

Usage notes

Migration limitations

• You can migrate data from only one bucket per migration task. Migrating data from an entire account at once is not supported.

• You cannot migrate buckets with metadata acceleration enabled.

• The following describes migration support for object attributes from Tencent COS:

  • Supported attributes: the LastModifyTime object attribute and the x-cos-meta-*, Content-Type, Cache-Control, Content-Encoding, Content-Disposition, Content-Language, and Expires headers.

  • Unsupported attributes include: StorageClass, ACL, server-side encryption, and Tagging.

    Note

    The preceding list is not exhaustive, and the migration behavior for unlisted attributes cannot be determined in advance. To confirm how an unlisted attribute is handled, migrate a small number of objects first and then check the result at the destination.

    Warning

    When you use a standard Tencent COS domain as the source, calls to the GetObject operation force the Content-Disposition header to be set to attachment. This header is migrated to the destination, which can cause browsers to forcibly download files like HTML pages or PNG images instead of displaying them. To prevent this behavior, use a custom domain as the source for your migration, as described in Prerequisites. This preserves the original Content-Disposition header.

Prerequisites

• Tencent Cloud API key: A Cloud API key is created for the Tencent Cloud account that owns the source data. Tencent COS uses this key to authenticate your identity and verify your read permissions on the source data. You enter this key in the AccessKeyId and SecretAccessKey fields when you create the source data address.

• Tencent Cloud APPID: The APPID of your Tencent Cloud account is part of the source bucket name.

  Note

  You can find the APPID on the Account Info page in the Tencent Cloud console.

• RAM role for the destination bucket: A role that grants Data Online Migration access to the destination OSS bucket is authorized. The authorization method depends on the account that owns the destination bucket. For more information, see the Authorize Role parameter in Step 3: Create a destination address.

• (Conditional) Custom domain for the source bucket: If you must retain the original Content-Disposition header of the source objects, bind a custom domain in the Tencent COS console and use that custom domain as the source for your migration.

Step 1: Select a region

1. Log in to the Data Online Migration console as the RAM user you created.

2. The region you select in the upper-left corner of the top navigation bar is the migration service deployment region. Therefore, select the region where your data source is located or the one closest to it.

   The region you select at the top of the console is the migration service deployment region. Available regions in the Chinese mainland include China (Beijing), China (Shanghai), China (Hangzhou), China (Shenzhen), and China (Ulanqab). Other available regions include China (Hong Kong), Singapore, Germany (Frankfurt), and US (Virginia).

   Important

   • Data endpoints and migration tasks are region-specific. Choose your region carefully.

   • We recommend selecting your source data's region. If this region is not available, select the closest available region.

   • For cross-border migrations, we recommend enabling transfer acceleration to improve migration speed. Buckets with transfer acceleration enabled incur transfer acceleration fees. For more information about transfer acceleration, see Accelerate access to OSS.

Step 2: Create a source address

1. In the navigation pane on the left, choose Data Online Migration > Address Management, and then click Create Address.

2. On the Create Address page, set the following parameters and click OK.

   Parameter	Required	Description
   Name	Yes	The name must be 3 to 63 characters in length. The name can contain lowercase letters, digits, hyphens (-), and underscores (_). The name is case-sensitive. The name cannot start with a hyphen (-) or an underscore (_).
   Type	Yes	Select Tencent COS.
   Domain name	Yes	Enter the endpoint for the bucket, such as cos.ap-nanjing.myqcloud.com. To retain the original Content-Disposition header of the migrated objects, enter a custom domain that is bound to the bucket instead, as described in Migration limitations. For more information about Tencent COS domain names, see Region and Access Endpoints.
   AccessKeyId	Yes	Enter the AccessKeyId of the Cloud API key that you created for the source account.
   SecretAccessKey	Yes	Enter the SecretAccessKey of the same Cloud API key.
   Bucket	Yes	Enter the name of the Tencent COS bucket. The bucket name must be in the custom-name-APPID format. For example, example-1234567890.
   Prefix	No	Specify a prefix to migrate a subset of the data. If specified, only data under that prefix is migrated. The prefix must not start with a forward slash (/) but must end with one. If you omit this parameter, all data in the bucket is migrated.
   Tunnel	No	Select the name of the channel to use. Important This parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system. You must associate an agent if the destination is a local file system or if the migration uses a dedicated connection, for example, to a financial cloud or Apsara Stack environment.
   Agent	No	Select the name of the agent to use. Important This parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system. You can select a maximum of 200 agents for a specified channel.

Step 3: Create a destination address

1. In the navigation pane on the left, choose Data Online Migration > Address Management, and then click Create Address.

2. On the Create Address panel, set the following parameters, and then click OK.

Parameter	Required	Description
Name	Yes	Enter a name for the destination data address. The name must meet the following requirements: The name must be 3 to 63 characters in length. The name can contain lowercase letters, digits, hyphens (-), and underscores (_). The name is case-sensitive. The name cannot start with a hyphen (-) or an underscore (_).
Type	Yes	Select OSS.
Custom domain name	No	A custom domain name mapped to the destination OSS bucket.
Region	Yes	Select the region of the destination data address. For example, China (Hangzhou).
Authorize Role	Yes	If the destination bucket is in your current account: Authorize a role in the Data Online Migration console (Recommended). Manually authorize a role in the Data Online Migration console (Alternative). If the destination bucket is in a different account: Authorize a role in the OSS console.
Bucket	Yes	Enter the name of the destination bucket for the migrated data.
Prefix	No	You can specify a prefix to migrate source data to a specific directory. The prefix cannot start with a forward slash (/) but must end with a forward slash (/). Example: data/to/oss/. Specify a prefix: For example, if the prefix of the source data address is example/src/, which contains a file named example.jpg, and you set the destination prefix to example/dest/, the full path of the migrated file example.jpg is example/dest/example.jpg. Do not specify a prefix: If you leave this parameter empty, the source data is migrated to the destination bucket's root directory.
Tunnel	No	Select the name of the channel to use. Important This parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system. You must associate an agent if the destination is a local file system or if the migration uses a dedicated connection, for example, to a financial cloud or Apsara Stack environment.
Agent	No	Select the name of the agent to use. Important This parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system. You can select a maximum of 200 agents for a specified channel.

Step 4: Create a migration task

1. In the left-side navigation pane, choose Data Online Migration > Migration Tasks, and then click Create Task.

2. On the Select Address page, configure the following parameters, and then click Next.

   Parameter	Required	Description
   Name	Yes	Enter a name for the migration task. The name must meet the following requirements: The name must be 3 to 63 characters in length. The name can contain lowercase letters, digits, hyphens (-), and underscores (_). The name is case-sensitive. The name cannot start with a hyphen (-) or an underscore (_).
   Source Address	Yes	Select an existing source address.
   Destination Address	Yes	Select an existing destination address.

3. On the Task Configurations page, configure the following parameters.

   Parameter	Required	Description
   Basic settings
   Migration Bandwidth	No	Select the migration bandwidth. Default: The maximum available bandwidth. The actual speed depends on the file size and the number of files. Specify an upper limit: Specify a maximum bandwidth limit. Important The actual migration bandwidth depends on factors such as the data source, network conditions, destination throttling, and file size, and may not reach the specified upper limit. Select a value based on your data source, destination, business requirements, and network bandwidth. Improper throttling can disrupt your business operations.
   Files Migrated Per Second	No	Select the number of files migrated per second. Default: Uses the default rate for files migrated per second. Specify an upper limit: Specify a maximum number of files migrated per second. Important The actual number of files migrated per second is affected by factors such as the data source, network conditions, destination throttling, and file size, and may not reach the specified upper limit. Select a value based on your data source, destination, business requirements, and network bandwidth. Improper throttling can disrupt your business operations.
   Overwrite Mode	Yes	Specifies how to handle files with the same name. Do not overwrite: Skips the file. Overwrite All: The source file overwrites the destination file. Overwrite based on the last modification time: If the last modified time of the source file is later than that of the destination file, the destination file is overwritten. If the last modified time of the source and destination files are the same, the destination file is overwritten if their Size or Content-Type differs. Warning With the Overwrite based on the last modification time policy, there is a risk that an older file may overwrite a newer file because it does not strictly guarantee that newer files will be preserved. If you select the Overwrite based on the last modification time policy, ensure that the source files can return metadata such as last modified time, Size, and Content-Type. Otherwise, the overwrite policy may not work as expected. When you select Do not overwrite or Overwrite if last modified time is later, the service sends one request to the source and one to the destination to retrieve metadata for the overwrite check. This incurs request fees at both the source and the destination.
   Audit settings
   Migration Report	Yes	Specifies how to push the migration report. Do not push (Default): Does not push the migration report to the destination bucket. Push: Pushes the migration report to the destination bucket. For path details, see Next steps. Important Pushing the migration report consumes storage space at the destination. Pushing the migration report may be delayed. Each task execution record has a unique ID. The migration report is pushed only once. Do not delete it unless necessary.
   Migration Logs	Yes	Specifies how to push the migration log. Do not push (Default): Does not push the migration log. Push: Pushes the migration log to Log Service (SLS). You can view the migration log in SLS. Push only file error logs.: Pushes only file error logs to Log Service (SLS). You can view these logs in SLS. When you select Push or Push only file error logs., Online Migration Service creates a project in Log Service (SLS) named aliyun-oss-import-log-<Alibaba Cloud account ID>-<current deployment region>, for example, aliyun-oss-import-log-137918634953****-cn-hangzhou. Important Before you select Push or Push only file error logs., ensure you have completed the following operations. Otherwise, the migration task may fail. You have activated Log Service (SLS). You have granted the required permissions on the Authorize page.
   Authorize	No	This option appears only when you set Migration Logs to Push or Push only file error logs.. Click Authorize to go to the Cloud Resource Access Authorization page. The system creates the AliyunOSSImportSlsAuditRole role and grants it the necessary permissions. Click Agree to Authorization to complete the authorization.
   Filters
   File Name	No	A filter based on filenames. Supports Include and Exclude rules using regular expressions based on the RE2 library (only a subset of the syntax is supported). For example: .*\.jpg$ matches all files that end with .jpg. ^file.* matches all files at the root level that start with file by default. If you set a prefix for the source address, such as data/to/oss/, you must use ^data/to/oss/file.* to match all files that start with file under that prefix. .*/picture/.* matches any subdirectory named picture. Important When you use an Include rule, all matching files are migrated. If there are multiple Include rules, any file that matches at least one rule is migrated. For example, if you have two files, picture.jpg and picture.png, and you set an Include rule for .*\.jpg$, only picture.jpg is migrated. If you also add an Include rule for .*\.png$, both files are migrated. When you use an Exclude rule, no matching files are migrated. If there are multiple Exclude rules, any file that matches at least one rule is not migrated. For example, if you have two files, picture.jpg and picture.png, and you set an Exclude rule for .*\.jpg$, only picture.png is migrated. If you also add an Exclude rule for .*\.png$, neither file is migrated. Exclude rules take precedence. If a file matches both an Exclude rule and an Include rule, the file is not migrated. For example, for a file named file.txt, if you set an Exclude rule for .*\.txt$ and an Include rule for file.*, the file file.txt is not migrated.
   File Modification Time	No	Filters files based on their last modified time. You can specify a time range to migrate only files whose last modified time falls within that range. The rules are as follows: If you specify only a start time of January 1, 2019, the task migrates only files last modified on or after that date. If you specify only an end time of January 1, 2022, the task migrates only files last modified on or before that date. If you specify a start time of January 1, 2019 and an end time of January 1, 2022, the task migrates only files last modified on or between these two dates.
   Migration configuration
   Retain last modified time	Yes	Specifies whether to retain the last modified time of the source file. Retain (Default): The destination file retains the last modified time of the source file. Do not retain: The last modified time is not set.
   Convert Appendable Object to Normal Object/Multipart Object	No	Specifies whether to convert Appendable Objects to Normal Objects or Multipart Objects. Yes: Converts source Appendable Objects to Normal Objects or Multipart Objects at the destination. No: Source Appendable Objects are migrated to the destination as Appendable Objects. Important This option is visible and configurable only if your account is on the allowlist. This option is currently supported only for OSS-to-OSS migration tasks.
   Specify destination storage class	No	Specifies whether to set the storage class for destination files. Specify: Migrated files are assigned the specified storage class at the destination. The following storage classes are available: Standard Infrequent Access Archive Cold Archive Deep Cold Archive Do not specify (Default): Migrated files at the destination use the default storage class of the destination bucket. Important This option is visible and configurable only if your account is on the allowlist. This option is currently supported only for tasks with an OSS destination.
   Task scheduling
   Execution time	No	Important If a migration task is still running at its next scheduled execution time, that execution is postponed until the next scheduled time after the current run completes. This continues until the specified number of executions is reached. Concurrency limit for migration tasks: Deployment regions in the Chinese mainland and China (Hong Kong) support a maximum of 10 concurrent tasks. Overseas regions support a maximum of 5 concurrent tasks. Exceeding this limit may cause scheduled tasks to fail. Defines when to run the migration task. Immediately: Runs the task as soon as it is created. At the Specified Time: Specifies a daily time window for the task to run. By default, the task starts at the specified start time and pauses at the specified stop time. Periodic Scheduling: Runs the task based on a specified frequency and number of executions. Execution frequency: Supports five types of frequencies: Hourly, Daily, Weekly, Specific days of the week, and Custom. For more information, see Execution frequency reference. Number of executions: Specifies how many times the task runs. If not set, the task runs once by default. For the maximum number of executions, refer to the console prompt. Important You can manually start and stop tasks at any time, regardless of the custom execution time settings.

4. Read the Online Migration Service Agreement, select I understand and confirm the compliance commitment statement and acknowledge my responsibility to verify data consistency after the migration task is complete, and then click Next.

5. Review your configuration settings. If everything is correct, click OK and wait for the migration task to run.

Execution frequency

Step 5: Verify data