This topic describes how to migrate data between Object Storage Service (OSS) buckets.

Note

If you want to perform an online migration, take note of the following items:
  • A migration job occupies the network resources of the source data address and the destination data address. To ensure business continuity, we recommend that you specify a rate limit for your migration job or run your migration job during off-peak hours.
  • Before a migration job starts, the system checks the files at the source data address and the destination data address. If a source file has the same name as a destination file and the source file is updated later than the destination file, the destination file is overwritten. If the two files have different content, you must change the name of one file or back up the destination file.
  • If symbolic links exist at the source data address, the files to which the symbolic links point are migrated. The destination files are named after the original symbolic links. For example, a symbolic link named a exists at Source Data Address A. The file to which the symbolic link points is b.jpg. During migration, b.jpg is migrated to the destination data address and renamed a. For more information about symbolic links, see Configure symbolic links.
    Note If the objects that you want to migrate contain symbolic links, the number of migration workloads may be greater than the estimated number and the migration progress may exceed 100%. You can evaluate the actual progress based on the size of the migrated files.
  • Data Online Migration allows you to migrate only the data of a bucket at a time. You cannot migrate all of the data that belongs to your account at the same time.

Step 1: Create a source data address

  1. Log on to the Alibaba Cloud Data Transport console.
  2. In the left-side navigation pane, choose Data Online Migration > Data Address. On the page that appears, click Create Data Address.
  3. In the Create Data Address panel, configure the parameters in the following table and click OK.
    Parameter Required Description
    Data Type Yes Select OSS.
    Data Region Yes Select the region where the source data address resides. For example, select China (Hangzhou).
    Data Name Yes Enter a name. The name must be 3 to 63 characters in length. Special characters are not supported, except for hyphens (-) and underscores (_).
    Enable Transfer Acceleration No Before you can enable the transfer acceleration feature of OSS for Data Online Migration, you must enable the transfer acceleration feature for buckets. The transfer acceleration feature takes effect within 30 minutes after the feature is enabled. If you enable the feature, wait for 30 minutes before you create a migration job.
    Notice You are charged for enabling the transfer acceleration feature for OSS buckets. For more information, see Transfer acceleration.
    OSS Endpoint Yes Select an endpoint based on the region where your data is stored. For more information, see Regions and endpoints.
    • http://oss-cn-endpoint.aliyuncs.com: An HTTP-based endpoint is used to access OSS over the Internet.
    • https://oss-cn-endpoint.aliyuncs.com: An HTTPS-based endpoint is used to access OSS over the Internet.
    • http://oss-cn-qingdao-internal.aliyuncs.com: An HTTP-based endpoint is used to access OSS over an internal network.
    • https://oss-cn-qingdao-internal.aliyuncs.com: An HTTP-based endpoint is used to access OSS over an internal network.
    Notice For example, an OSS data address uses an internal endpoint and is used as the source data address when you create a migration job. In this case, the destination data address can be only an Apsara File Storage NAS (NAS) or OSS data address in the same region.
    Access Key Id and Access Key Secret Yes Specify the AccessKey pair that you can use to migrate data. For more information, see Create a RAM user and grant permissions.
    OSS Bucket Yes Select the bucket that stores the data that you want to migrate.
    OSS Prefix Yes An OSS prefix cannot start with a forward slash (/) and must end with a forward slash (/). For example, you can specify data/to/oss/ as the OSS prefix.
  4. Optional:If you are prompted that this feature is in public preview, click Application to apply for whitelist permission.
  5. Enter the required information and submit the application for public preview. After the application is approved, you will receive a text message.

Step 2: Create a destination data address

You can create a destination data address the same way you create a source data address. For information about how to configure the parameters, see Step 1.

Step 3: Create a migration job

  1. In the left-side navigation pane, choose Data Online Migration > Migration Jobs. On the page that appears, click Create Job.
  2. In the Create Job panel, read the Terms of Migration Service and select I understand the above terms and conditions, and apply for opening data migration service. Click Next.
    The Fee Reminder dialog box appears, indicating that you are charged for your OSS requests. Click Yes, Go Ahead. OSS request fee
  3. On the Job Config tab, configure the parameters in the following table and click Next.
    Parameter Required Description
    Job Name Yes Enter a job name. The name must be 3 to 63 characters in length, and can contain only lowercase letters, digits, and hyphens (-). It cannot start or end with a hyphen (-).
    Source Data Address Yes Select the source data address that you created.
    Destination Data Address Yes Select the destination data address that you created.
    Notice The source data address and the destination data address may be located in different countries. In this case, before you configure the migration job, you must submit a ticket to request permissions to create migration jobs across countries. You must guarantee that your business is legitimate, data transit complies with local laws and regulations, and your data does not contain illegal information.
    Specified Directory No
    • Do not filter: All the data at the source data address is migrated.
    • Exclude: The files and subdirectories in the excluded directories are not migrated.
    • Contain: Only the files and subdirectories in the specified directories are migrated.
    Note
    • A directory cannot start with a forward slash (/) or a backslash (\), and cannot contain double forward slashes (//), two periods (..),or double quotation marks ("). The character string that consists of all the specified directory names cannot exceed 10 KB.
    • A directory must end with a forward slash (/). For example, you can specify docs/ as the directory.
    • You can specify a maximum of 20 directories of the Exclude or Contain type.
    Migration Type Yes
    • Full: The system runs a full migration job. You must specify the Start Time Point of File parameter. The system migrates the data whose last modification time is later than the specified start time at a time. After all data is migrated, the migration job automatically stops. You can submit the job again if the data at the source data address changes. When you submit the job again, the system only migrates the data that is changed after the previous job.
    • Incremental: The system runs the migration job based on the specified Migration Interval and Migration Times parameters. You must specify the Start Time Point of File parameter. During the first migration, the system migrates the full data that is modified after the specified start time. After the first migration is completed, incremental migrations are performed based on the specified Migration Interval parameter. For each incremental migration, the system migrates only the data that is created or modified after the previous migration started and before the current migration starts. If you set Migration Times to N, full migration is performed once and then incremental migration is performed (N - 1) times. For example, you set Migration Interval to 1 hour, Migration Times to 5, and Start Time Point of File to 08:00:00, March 5, 2019. The current time is 08:00, March 10, 2019. When the first migration starts, the system migrates the files that are last modified between 08:00:00, March 5, 2019 and 08:00:00, March 10, 2019. In this example, the first migration requires 1 hour to complete and the second migration starts at 10:00:00, March 10, 2019. The two hours from 8:00:00 to 10:00:00 include the 1 hour that the first migration requires and the migration interval of 1 hour. During the second migration, the system migrates the files that are last modified between 08:00:00, March 10, 2019 and 10:00:00, March 10, 2019. The migration job includes one full migration and four incremental migrations.
    Notice Before you start a full migration job or an incremental migration job, the system compares the files at the source data address with those at the destination data address. If a source file has the same name as a destination file, the destination file is overwritten if one of the following conditions is met:
    • The Content-Type headers of the source file and the destination file are different.
    • The source file is updated after the previous migration.
    • The size of the source file is different from the size of the destination file.
    Start Time Point of File Yes
    • All: All files are migrated.
    • Assign: Only the files that are created or modified after the specified time are migrated. For example, if you set the start time to 08:00:00, November 01, 2018, only the files that are created or modified after 08:00:00, November 01, 2018 are migrated. The files that are created or modified before the specified time are skipped.
    Migration Interval Yes. Only for incremental migration. The default value is 1 hour and the maximum value is 24 hours.
    Migration Times Yes. Only for incremental migration. The default value is 1 and the maximum value is 30.
    File Overwrite Method Yes The method used to process a source file that has the same name as a destination file during migration. Valid values:
    • LastModified: If a source file has the same name as a destination file, the system compares the values of this header for the files.
      • If the value of the LastModified header for the source file is smaller than the value of the header for the destination file, the source file is not migrated and the destination file is not overwritten.
      • If the value of the LastModified header for the source file is greater than the value of the header for the destination file, the destination file is overwritten.
      • If the values of the LastModified header for the files are equal, the system checks whether the values of the Size and Content-Type headers for the files are equal.

        If the values of the Size and Content-Type headers for the files are equal, the source file is not migrated and the destination file is not overwritten. If the values of one or both headers are different between the files, the destination file is overwritten.

    • Condition: If a source file has the same name as a destination file, the system compares the values of the LastModified, Size, and Content-Type headers for the files.
      • If the values of the LastModified, Size, and Content-Type headers for the files are equal, the source file is not migrated and the destination file is not overwritten.
      • If the values of at least one of the LastModified, Size, and Content-Type headers are different between the files, the destination file is overwritten.
    • All: If a source file has the same name as a destination file, the destination file is overwritten and condition checks are not performed.
    • No: If a source file has the same name as a destination file, the source file is not migrated and the destination file is not overwritten.
  4. On the Performance tab, navigate to the Data Prediction section and specify the Data Size and File Count parameters.
    Note To ensure a successful migration, we recommend that you estimate the amount of data to be migrated as accurately as possible. For more information, see Estimate the amount of data to be migrated.
  5. Optional. On the Performance tab, navigate to the Flow Control section, specify the Time Range and Max Flow parameters, and then click Add.
    Note To ensure business continuity, we recommend that you specify the Time Range and Max Flow parameters based on traffic peaks and troughs.
  6. Click Create and wait until the migration job is completed.