Data Online Migration:Migrate data

Usage notes

When you migrate data by using Data Online Migration, take note of the following items:

• Data Online Migration accesses the source data by using the public interfaces provided by the storage service provider of the source data. The access behavior depends on the interface implementation of the storage service provider.

• When Data Online Migration is used for migration, it consumes resources at the source and destination data addresses. This may affect your business. To ensure business continuity, we recommend that you enable throttling for your migration tasks or run them during off-peak hours after careful assessment.

• Before a migration task starts, Data Online Migration checks the files at the source data address and the destination data address. If a file at the source data address and a file at the destination data address have the same name, and the File Overwrite Method parameter of the migration task is set to Yes, the file at the destination data address is overwritten during migration. If the two files contain different information and the file at the destination data address needs to be retained, we recommend that you change the name of one file or back up the file at the destination data address.

• The LastModified property of the source file is retained after the file is migrated to the destination bucket. If a lifecycle rule is configured for the destination bucket and takes effect, the migrated file whose last modification time is within the specified time period of the lifecycle rule may be deleted or archived in specific storage types.

Limits

Data Online Migration allows you to migrate only the data of a single bucket per job. You cannot migrate all data that belongs to your account in a single job.

Step 1: Create a source data address

1. Log on to the Data Online Migration console as the Resource Access Management (RAM) user that you created for data migration.

2. In the left-side navigation pane, choose Data Online Migration > Data Address. On the page that appears, click Create Data Address.

3. In the Create Data Address panel, configure the parameters and click OK. The following table describes the parameters.

   Parameter	Required	Description
   Data Type	Yes	Select AWS S3 Inventory.
   Data Name	Yes	The name of the source data address. The name must be 3 to 63 characters in length. The name cannot contain special characters except for hyphens (-) and underscores (_).
   Endpoint	Yes	Enter the endpoint for accessing AWS Simple Storage Service (S3). For more information, see Regions and endpoints.
   Bucket	Yes	The name of the Amazon S3 bucket in which the data to be migrated is stored. Note The bucket name cannot contain invalid characters, such as spaces, line breaks, or tab characters.
   Access Key Id	Yes	Specify the key pair that is used to access the AWS S3 bucket. After the migration is complete, delete the key pair.
   Secret Access Key	Yes
   Inventory Endpoint	Yes	Enter the endpoint for accessing AWS S3 Inventory.
   Inventory AccessKey Id	Yes	Specify the key pair that is used to access AWS S3 Inventory. After the migration is complete, delete the key pair.
   Inventory Secret AccessKey	Yes
   InventoryAddress	Yes	Enter the URL of the AWS S3 inventory lists, such as s3://{bucket}/{ObjectKey}. In this example, bucket indicates the name of the bucket where the inventory lists are located, and ObjectKey is the path of the manifest.json file.

Step 2: Create a destination data address

1. In the left-side navigation pane, choose Data Online Migration > Data Address. On the page that appears, click Create Data Address.

2. In the Create Data Address panel, configure the parameters and click OK. The following table describes the parameters.

   Parameter	Required	Description
   Data Type	Yes	Select OSS.
   Data Name	Yes	The name of the source data address. The name must be 3 to 63 characters in length. The name cannot contain special characters except for hyphens (-) and underscores (_).
   Data Region	Yes	The region in which the destination data address resides.
   Enable Transfer Acceleration	No	If you want to use Data Online Migration together with the transfer acceleration feature of Object Storage Service (OSS), enable the transfer acceleration feature for buckets. The transfer acceleration feature takes effect within 30 minutes after it is enabled. After you enable the feature, wait for 30 minutes before you create a migration job. Important If you use the transfer acceleration feature to access OSS buckets, you are charged additional transfer acceleration fees. For more information, see Enable transfer acceleration.
   OSS Endpoint	Yes	The endpoint of the source data address. Select or enter an endpoint based on the region in which the source data address resides. For more information, see Regions and endpoints. Note When you migrate data from third-party cloud services to OSS, you must access OSS from a public endpoint over the Internet.
   AccessKey Id	Yes	The AccessKey pair that you want to use to migrate data. For more information, see the Create a Resource Access Management (RAM) user and grant required permissions to the RAM user section of the "Before you begin" topic. Note If you want to migrate data across accounts, you must specify the AccessKey pair of the source Alibaba Cloud account.
   AccessKey Secret	Yes
   OSS Bucket	Yes	The OSS bucket in which the data to be migrated is stored.
   OSS Prefix	No	The prefix of the destination data address. The value cannot start with a forward slash (/) and must end with a forward slash (/). Example: data/to/oss/. If you want to migrate data to the root directory of the specified bucket, leave this parameter empty. Important If the name of a source file starts with a forward slash (/), you must specify an OSS prefix when you specify the destination data address. If no OSS prefix is specified, the migration job fails. For example, if the name of a file that you want to migrate is /test/test.png, you must specify an OSS prefix, such as oss/. After the migration is complete, the name of the file changes from /test/test.png to oss//test/test.png.

Step 3: Create a migration job

1. In the left-side navigation pane, choose Data Online Migration > Migration Jobs. On the page that appears, click Create Job.

2. In the Create Job panel, read the terms of the migration service and select I agree to the above terms and activate the Data Transport service. Then, click Next.

3. In the Fee Reminder message, click Yes, Go Ahead.

4. In the Job Config step, configure the following parameters and click Next.

   Parameter	Required	Description
   Job Name	Yes	Enter a job name. The name must be 3 to 63 characters in length and can contain only lowercase letters, digits, and hyphens (-). It cannot start or end with a hyphen (-).
   Source Data Address	Yes	Select the source data address that you created.
   Destination Data Address	Yes	Select the destination data address that you created.
   Specified Directory	No	Specify the directories whose files and subdirectories are included or excluded during migration. Valid values: Do not filter: migrates all data from the source data address. Exclude: does not migrate the files or subdirectories in the specified directories. For example, you want to migrate all the directories in the root_dir/ directory except root_dir/bad_sub1/ and root_dir/bad_sub2/. In this case, you can select this method and specify bad_sub1/ and bad_sub2/. Contain: migrates only the files and subdirectories in the specified directories. For example, you want to migrate only root_dir/good_sub1/ and root_dir/good_sub2/ in the root_dir/ directory. In this case, you can select this method and specify good_sub1/ and good_sub2/. Note A directory name cannot start with a forward slash (/) or a backslash (\), and cannot contain double forward slashes (//), double periods (..), or double quotation marks ("). The character string that consists of all the specified directory names cannot exceed 10 KB in length. A directory name must end with a forward slash (/). Example: docs/. You can specify up to 20 directories that you want to include or exclude.
   Migration Type	Yes	Full: The system runs a full migration job. After all data is migrated, a migration job ends. If the data at the source data address changes after the migration is complete, you can submit another job to migrate all data. In this case, the system migrates only the data that changes. Important Before you start a full migration job, the system compares files at the source data address with those at the destination data address. If a file at the destination data address has the same name as a file at the source data address, the file at the destination data address is overwritten when one of the following conditions is met: The Content-Type property values of the source file and the destination file are different. The last modification time of the source file is later than that of the destination file. The size of the source file is different from the size of the destination file.
   File Overwrite Method	Yes	Specify the method that is used to process the source file whose name is the same as a destination file. Valid values: LastModified: If a file at the source data address has the same name as a file at the destination data address, the system compares the last modification time that is indicated by the LastModified property of the files. If the last modification time of the file at the source data address is later than that of the file at the destination data address, the file at the source data address is migrated and the file at the destination data address is overwritten. If the last modification time of the file at the source data address is earlier than that of the file at the destination data address, the file at the source data address is not migrated and the file at the destination data address is retained. If the two files have the same last modification time, the system checks the values of the Size and Content-Type properties of the files. If the two files have the same Size and Content-Type values, the file at the source data address is not migrated. If the two files have different values of at least one property, the file at the source data address is migrated and the file at the destination data address is overwritten. Condition: If a file at the source data address has the same name as a file at the destination data address, the system compares the values of the LastModified, Size, and Content-Type properties of the two files. If all property values of the two files are the same, the file at the source data address is not migrated. If the two files have different values of at least one property, the file at the source data address is migrated and the file at the destination data address is overwritten. All: If a file at the source data address has the same name as a file at the destination data address, the system performs no comparison and directly overwrites the file at the destination address with the file at the source data address. No: If a file at the source data address has the same name as a file at the destination data address, the system performs no comparison and directly skips the file at the source data address during the migration.

5. In the Data Prediction section of the Performance step, configure the Data Size and File Count parameters.

   Note

   To ensure a successful migration job, we recommend that you estimate the volume of data that you want to migrate as accurately as possible. For more information, see the Estimate the amount of data that you want to migrate section of the "Before you begin" topic.

6. Optional. In the Flow Control section of the Performance step, configure the Time Range and Max Flow(MB/s) parameters and click Add.

   Note

   To ensure business continuity, we recommend that you configure the Time Range and Max Flow(MB/s) parameters based on the workload peaks and troughs.

7. Click Create and wait until the migration job is complete.

Step 4: Verify data