Migrate data from an HTTP or HTTPS source to an OSS bucket by creating source and destination addresses and configuring a migration task.
Precautions
When you use Data Online Migration:
The online migration service accesses source data using the standard public APIs of the source storage service provider. The migration behavior depends on the provider's service implementation.
Online migration consumes resources at both the source and the destination, potentially impacting your business operations. If your services are critical, evaluate the potential impact and configure a speed limit for the migration task, or run the task during off-peak hours.
The online migration service checks files at both the source and destination before a migration begins. However, if a file at the source has the same name as a file at the destination and the migration task is configured to overwrite existing files, the service overwrites the destination file. If the files contain different data, rename one of the files or back up the destination file before starting the migration.
The online migration service preserves the last modified time of source files. If a lifecycle rule is configured for the destination bucket, a migrated file whose last modified time matches the rule's conditions may be deleted or transitioned to the specified archive storage class.
Migration limitations
Object attributes during HTTP/HTTPS migration:
-
These attributes are preserved:
LastModifyTime,Content-Type,Cache-Control,Content-Encoding,Content-Disposition,Content-Language, andExpires. -
Other attributes may not be migrated. Actual values are determined at migration time.
Step 1: Select a region
-
Log in to the Data Online Migration console as the RAM user you created.
The region you select in the upper-left corner of the top navigation bar is the migration service deployment region. Therefore, select the region where your data source is located or the one closest to it.
The region you select at the top of the console is the migration service deployment region. Available regions in the Chinese mainland include China (Beijing), China (Shanghai), China (Hangzhou), China (Shenzhen), and China (Ulanqab). Other available regions include China (Hong Kong), Singapore, Germany (Frankfurt), and US (Virginia).
ImportantData endpoints and migration tasks are region-specific. Choose your region carefully.
We recommend selecting your source data's region. If this region is not available, select the closest available region.
For cross-border migrations, we recommend enabling transfer acceleration to improve migration speed. Buckets with transfer acceleration enabled incur transfer acceleration fees. For more information about transfer acceleration, see Accelerate access to OSS.
Step 2: Create a source address
In the navigation pane on the left, choose Data Online Migration > Address Management, and then click Create Address.
-
In the Create Address panel, set the following parameters and click OK.
-
Parameter
Required
Description
Name
Yes
The name must be 3 to 63 characters in length.
The name can contain lowercase letters, digits, hyphens (-), and underscores (_). The name is case-sensitive.
The name cannot start with a hyphen (-) or an underscore (_).
Type
Yes
Select HTTP/HTTPS.
Protocol
Yes
Select HTTP or HTTPS.
Port
Yes
Enter the port number. Default: 80 (HTTP) or 443 (HTTPS). Valid values: 0 to 65535.
Inventory Location
Yes
Select the service that stores the inventory file: Alibaba Cloud OSS or AWS S3.
Inventory Path
Yes
Enter the path to the manifest.json file.
Inventory Domain Name or Region
Yes
-
If you set the Inventory Location parameter to Alibaba OSS, specify the region in which the OSS inventory list resides.
-
If Inventory Location is set to AWS S3, enter the domain name used to access the AWS S3 inventory. For more information, see Amazon S3 endpoint.
Role
Yes (if Inventory Location is Alibaba Cloud OSS)
-
If the inventory bucket is in your current account:
-
If the inventory bucket is in a different account:
Bucket
Yes
Enter the inventory bucket name. Must be in your current Alibaba Cloud account.
Inventory AccessKeyId
Yes (if Inventory Location is AWS S3)
If Inventory Location is set to AWS S3, enter the access key used to access the AWS S3 inventory list. For security, delete this key after the migration is complete.
Inventory SecretAccessKey
Tunnel
No
Select the name of the channel to use.
ImportantThis parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system.
You must associate an agent if the destination is a local file system or if the migration uses a dedicated connection, for example, to a financial cloud or Apsara Stack environment.
Agent
No
Select the name of the agent to use.
ImportantThis parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system.
You can select a maximum of 200 agents for a specified channel.
Step 3: Create a destination address
In the navigation pane on the left, choose Data Online Migration > Address Management, and then click Create Address.
-
In the Create Address panel, set the following parameters and click OK.
The name must be 3 to 63 characters in length.
The name can contain lowercase letters, digits, hyphens (-), and underscores (_). The name is case-sensitive.
The name cannot start with a hyphen (-) or an underscore (_).
-
If the destination bucket is in your current account:
-
If the destination bucket is in a different account:
-
With prefix: If the source prefix is
example/src/and contains example.jpg, setting the destination prefix toexample/dest/migrates the file toexample/dest/example.jpg. -
Without prefix: Data is migrated to the bucket root directory.
This parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system.
You must associate an agent if the destination is a local file system or if the migration uses a dedicated connection, for example, to a financial cloud or Apsara Stack environment.
This parameter is required only when migrating data over an Express Connect or VPN connection, or when migrating data from a self-managed storage system.
You can select a maximum of 200 agents for a specified channel.
|
Parameter |
Required |
Description |
|
Name |
Yes |
Enter a destination address name. The name must meet these requirements: |
|
Type |
Yes |
Select Alibaba OSS. |
|
Custom domain name |
No |
Optional custom domain name for accessing the destination OSS bucket. |
|
Region |
Yes |
Select the destination bucket region, such as China (Hangzhou). |
|
Authorize Role |
Yes |
|
|
Bucket |
Yes |
Enter the destination bucket name. Must be in your current Alibaba Cloud account. |
|
Prefix |
No |
Specify a prefix to migrate data to a specific directory. The prefix cannot start with a forward slash (/) but must end with one. Example: |
|
Tunnel |
No |
Select the name of the channel to use. Important |
|
Agent |
No |
Select the name of the agent to use. Important |
Step 4: Create a migration task
In the left-side navigation pane, choose Data Online Migration > Migration Tasks, and then click Create Task.
On the Select Address page, configure the following parameters, and then click Next.
Parameter
Required
Description
Name
Yes
Enter a name for the migration task. The name must meet the following requirements:
The name must be 3 to 63 characters in length.
The name can contain lowercase letters, digits, hyphens (-), and underscores (_). The name is case-sensitive.
The name cannot start with a hyphen (-) or an underscore (_).
Source Address
Yes
Select an existing source address.
Destination Address
Yes
Select an existing destination address.
-
On the Task Configurations page, set the following parameters.
Parameter
Required
Description
Basic settings
Migration Bandwidth
No
Select the migration bandwidth.
Default: The maximum available bandwidth. The actual speed depends on the file size and the number of files.
Specify an upper limit: Specify a maximum bandwidth limit.
ImportantThe actual migration bandwidth depends on factors such as the data source, network conditions, destination throttling, and file size, and may not reach the specified upper limit.
Select a value based on your data source, destination, business requirements, and network bandwidth. Improper throttling can disrupt your business operations.
Files Migrated Per Second
No
Select the number of files migrated per second.
Default: Uses the default rate for files migrated per second.
Specify an upper limit: Specify a maximum number of files migrated per second.
ImportantThe actual number of files migrated per second is affected by factors such as the data source, network conditions, destination throttling, and file size, and may not reach the specified upper limit.
Select a value based on your data source, destination, business requirements, and network bandwidth. Improper throttling can disrupt your business operations.
Overwrite Mode
No
Specifies how to handle files with the same name.
Do not overwrite: Skips the file.
Overwrite All: The source file overwrites the destination file.
Overwrite based on the last modification time:
If the last modified time of the source file is later than that of the destination file, the destination file is overwritten.
If the last modified time of the source and destination files are the same, the destination file is overwritten if their Size or Content-Type differs.
With the Overwrite based on the last modification time policy, there is a risk that an older file may overwrite a newer file because it does not strictly guarantee that newer files will be preserved.
If you select the Overwrite based on the last modification time policy, ensure that the source files can return metadata such as last modified time, Size, and Content-Type. Otherwise, the overwrite policy may not work as expected.
When you select Do not overwrite or Overwrite if last modified time is later, the service sends one request to the source and one to the destination to retrieve metadata for the overwrite check. This incurs request fees at both the source and the destination.
WarningAuditing
Migration Report
Yes
Delivery method for the migration report.
-
Do not push (Default): The report is not delivered to the destination bucket.
-
Push: The report is delivered to the destination bucket. Next steps.
Important-
Reports consume storage space at the destination.
-
Report delivery may be delayed.
-
Each execution has a unique ID. A report is pushed once per execution. Use caution when deleting reports.
Migration Logs
Yes
Specifies how to push the migration log.
Do not push (Default): Does not push the migration log.
Push: Pushes the migration log to Log Service (SLS). You can view the migration log in SLS.
Push only file error logs.: Pushes only file error logs to Log Service (SLS). You can view these logs in SLS.
When you select Push or Push only file error logs., Online Migration Service creates a project in Log Service (SLS) named aliyun-oss-import-log-<Alibaba Cloud account ID>-<current deployment region>, for example, aliyun-oss-import-log-137918634953****-cn-hangzhou.
ImportantBefore you select Push or Push only file error logs., ensure you have completed the following operations. Otherwise, the migration task may fail.
You have activated Log Service (SLS).
You have granted the required permissions on the Authorize page.
Authorize
No
This option appears only when you set Migration Logs to Push or Push only file error logs..
Click Authorize to go to the Cloud Resource Access Authorization page. The system creates the AliyunOSSImportSlsAuditRole role and grants it the necessary permissions. Click Agree to Authorization to complete the authorization.
Migration configuration
Preserve file last modified time
Yes
Preserve the last modified time of source files.
-
Preserve (Default): Applies the source file's last modified time to the destination object.
-
Do not preserve: The last modified time is not set.
Specify storage class for destination objects
No
Set a storage class for destination objects.
-
Specify: Assign a storage class to migrated objects:
-
Standard
-
Infrequent Access
-
Archive
-
Cold Archive
-
Deep Cold Archive
-
-
Do not specify (Default): Objects use the default storage class of the destination bucket.
Important-
Available only after your account is added to the allowlist.
-
Supported only for migration tasks where the destination is OSS.
Task scheduling
Execution time
No
ImportantIf a migration task is still running at its next scheduled execution time, that execution is postponed until the next scheduled time after the current run completes. This continues until the specified number of executions is reached.
Concurrency limit for migration tasks: Deployment regions in the Chinese mainland and China (Hong Kong) support a maximum of 10 concurrent tasks. Overseas regions support a maximum of 5 concurrent tasks. Exceeding this limit may cause scheduled tasks to fail.
Defines when to run the migration task.
Immediately: Runs the task as soon as it is created.
At the Specified Time: Specifies a daily time window for the task to run. By default, the task starts at the specified start time and pauses at the specified stop time.
Periodic Scheduling: Runs the task based on a specified frequency and number of executions.
Execution frequency: Supports five types of frequencies: Hourly, Daily, Weekly, Specific days of the week, and Custom. For more information, see Execution frequency reference.
Number of executions: Specifies how many times the task runs. If not set, the task runs once by default. For the maximum number of executions, refer to the console prompt.
ImportantYou can manually start and stop tasks at any time, regardless of the custom execution time settings.
Read the Online Migration Service Agreement, select I understand and confirm the compliance commitment statement and acknowledge my responsibility to verify data consistency after the migration task is complete, and then click Next.
Review your configuration settings. If everything is correct, click OK and wait for the migration task to run.
Execution frequency
Execution frequency | Description | Example |
Hourly | Run the task once every hour. You can use this option with the maximum number of runs. | The current time is 8:05. The frequency is set to hourly with a maximum of 3 runs. The first run starts at the next hour, 9:00.
|
Daily | Run the task once a day. You must specify an hour (0-23) for the task to start. You can use this option with the maximum number of runs. | The current time is 8:05. The task is scheduled to run daily at 10:00, with a maximum of 5 runs. The first run starts at 10:00 today.
|
Weekly | Run the task once a week. You must specify a day of the week and an hour (0-23) for the task to start. You can use this option with the maximum number of runs. | The current time is Monday, 8:05. The task is scheduled to run every Monday at 10:00, with a maximum of 10 runs. The first run starts at 10:00 today.
|
Specific days of the week | Run the task on selected days of the week. You must specify the days and an hour (0-23) for the task to start. | The current time is Wednesday, 8:05. The task is scheduled to run on Mondays, Wednesdays, and Fridays at 10:00. The first run starts at 10:00 today.
|
Custom | Use a cron expression to define a custom schedule for the task start time. | Note A cron expression consists of six space-separated fields that define the execution schedule: second, minute, hour, day of the month, month, and day of the week. The minimum interval is 1 hour. The following cron expression examples are for reference only. For more options, use a cron expression generator.
|
Step 5: Verify data
The migration service only transfers your data and does not guarantee data consistency or integrity. After the migration task completes, you must verify data consistency between the source and the destination.
After the migration task completes, you must verify the data at the destination. Verify that all of your data has been completely migrated to the destination before deleting the source data. If you delete the source data without verifying it, you are responsible for any resulting data loss.