This topic describes the notes and procedure for data migration.

Precautions

When you perform an online migration, take note of the following issues:
  • Data Online Migration uses the standard and publicly available API operations provided by the storage service provider of the source data address to access the source data. The data accessed by Data Online Migration depends on the specific API operations that are implemented by the service provider.
  • A migration job occupies the network resources of the source and destination data addresses. To ensure business continuity, we recommend that you specify a rate limit for your migration job or run your migration job during off-peak hours.
  • Before a migration job starts, the system checks the files at the source data address and the destination data address. If the source and destination data addresses have files with the same name, and the last modification time of the source file is later than the last modification time of the destination file, the destination file is overwritten. If both files contain different information, we recommend that you change the name of one file or back up the destination file.
  • Data Online Migration allows you to migrate only the data of a single bucket per job. You cannot migrate all data that belongs to your account at a time.
Warning Make sure that you verify the migrated data at the destination data address after the migration task is complete to ensure that the migrated data is correct. If you delete the data at the source data address before you verify the migrated data at the destination data address, you are liable for the losses and consequences caused by any data loss.

Step 1: Create a source data address

  1. Log on to the Alibaba Cloud Data Transport console.
  2. In the left-side navigation pane, choose Data Online Migration > Data Address. On the page that appears, click Create Data Address.
  3. In the Create Data Address panel, configure the parameters and click OK.
    ParameterRequiredDescription
    Data TypeYesSelect YOU PAI.
    Data NameYesEnter a name that is 3 to 63 characters in length. Special characters are not supported, except for hyphens (-) and underscores (_).
    Domain AddressYesThe endpoint of an Object Storage Service (OSS) SDK or the domain name in a requested URL when you call a RESTful API operation.
    You can use one of the following domain names:
    • http://v0.api.upyun.com (recommended for intelligent routing)
    • http://v1.api.upyun.com (China Telecom)
    • http://v2.api.upyun.com (China Netcom)
    • http://v3.api.upyun.com (China Mobile Tietong)
      Important UPYUN implements traffic throttling when you download large amounts of data. When you migrate large amounts of data, we recommend that you contact UPYUN customer service to remove the throttling policy. Alternatively, you can set the Use CDN parameter to migrate data based on the CDN domain.
    Use CDNNoSpecify whether to use a CDN domain to migrate data.
    • Do not use: A CDN domain is not used. The endpoint that is specified in the Domain Address parameter is used to migrate data.
    • Use: Enter the CDN domain name that is configured for your bucket. The specified CDN domain is used to migrate data in the migration job.
      Note
      • You can use a CDN domain to increase the migration rate and avoid the impact of traffic throttling that is implemented by the UPYUN origin.
      • If you want to use an HTTPS-based domain name, you must enable the HTTPS protocol for the UPYUN CDN service. For more information, see Manage HTTPS settings.
    Service NameYesYou can log on to the UPYUN Cloud console to view the service name on the UPYUN Storage Service page.
    Migration FolderYes
    • Migrate All Data: All the data in the bucket is migrated.

      When you migrate all data, you do not need to specify a folder.

    • Migrate Partial Data: Only the files in the directory that is specified by the prefix are migrated. The prefix must end with a forward slash (/). For example, you can specify data/to/oss/ as a prefix.
    Operator Name and Operator SecretYes
    1. Log on to the UPYUN Cloud console. On the Account Management page, click Create Operator to create an operator for data migration. You must grant the read permission to the operator.
    2. On the UPYUN Storage Service page, click the name of a bucket. On the configuration page, select Content Management and click Authorize to authorize the operator.

Step 2: Create a destination data address

  1. Choose Data Online Migration > Data Address, and click Create Data Address.
  2. In the Create Data Address panel, configure the parameters and click OK.
    ParameterRequiredDescription
    Data TypeYesSelect OSS.
    Data NameYesEnter a name that is 3 to 63 characters in length. Special characters are not supported, except for hyphens (-) and underscores (_).
    Data RegionYesSelect the region where the destination data address is located.
    Enable Transfer AccelerationNoIf you want to use Data Online Migration together with the transfer acceleration feature of Object Storage Service (OSS), enable the transfer acceleration feature for buckets. It takes less than 30 minutes for the transfer acceleration feature to take effect. After you enable the feature, wait for 30 minutes before you create a migration task.
    Important If you use the transfer acceleration feature to access your OSS buckets, you are charged additional transfer acceleration fees. For more information, see Enable transfer acceleration.
    OSS EndpointYes
    Select an endpoint based on the region where your data is located. For more information, see Regions and endpoints.
    • http://oss-cn-endpoint.aliyuncs.com: indicates that you use an HTTP-based endpoint to access OSS over the Internet.
    • https://oss-cn-endpoint.aliyuncs.com: indicates that you use an HTTPS-based endpoint to access OSS over the Internet.
    Note When you migrate third-party data to OSS, you must access OSS from an Internet endpoint.
    Access Key Id and Access Key SecretYesSpecify the AccessKey pair that is used to migrate data. For more information, see Create an AccessKey pair.
    OSS BucketYesSelect a bucket to store the migrated data.
    OSS PrefixNoAn OSS prefix cannot start with a forward slash (/) and must end with a forward slash (/). For example, you can specify data/to/oss/ as the OSS prefix. Do not specify this parameter if you want to migrate data to the root directory of the specified bucket.
    Important If the name of a source file starts with a forward slash (/), you must specify an OSS prefix when you configure the destination data address. If no OSS prefix is specified, the migration job fails. For example, if the name of a file to be migrated is /test/test.png, you must specify an OSS prefix, such as oss/. After the migration job is completed, the name of the OSS file changes from /test/test.png to oss//test/test.png.

Step 3: Create a migration job

  1. In the left-side navigation pane, choose Data Online Migration > Migration Jobs. On the page that appears, click Create Job.
  2. In the Create Job panel, read the terms of the migration service and select I agree to the above terms and activate the Data Transport service. Then, click Next.
  3. In the Fee Reminder message, click Yes, Go Ahead.
  4. In the Job Config step, configure the parameters and click Next. The following table describes the parameters.
    ParameterRequiredDescription
    Job NameYesEnter a job name. The name must be 3 to 63 characters in length and can contain only lowercase letters, digits, and hyphens (-). It cannot start or end with a hyphen (-).
    Source Data AddressYesSelect the source data address that you created.
    Destination Data AddressYesSelect the destination data address that you created.
    Specified DirectoryNoSpecify the directories in which the files and subdirectories are migrated or not migrated. Valid values:
    • Do not filter: migrates all data from the source data address.
    • Exclude: does not migrate the files or subdirectories in the specified directories.

      For example, you want to migrate all the directories in the root_dir/ directory except root_dir/bad_sub1/ and root_dir/bad_sub2/. In this case, you can select this method and specify bad_sub1/ and bad_sub2/.

    • Contain: migrates only the files and subdirectories in the specified directories.

      For example, you want to migrate only root_dir/good_sub1/ and root_dir/good_sub2/ in the root_dir/ directory. In this case, you can select this method and specify good_sub1/ and good_sub2/.

    Note
    • A directory name cannot start with a forward slash (/) or a backslash (\), and cannot contain double forward slashes (//), double periods (..), or double quotation marks ("). The character string that consists of all the specified directory names cannot exceed 10 KB in length.
    • A directory name must end with a forward slash (/). For example, you can specify docs/ as a directory name.
    • You can specify up to 20 directories that you want to include or exclude.
    Migration TypeYesSelect the data migration mode. Valid values:
    • Full: The system migrates all data from the source data address to the destination data address. You must configure the Start Time Point of File parameter. The system migrates all data whose last modification time is later than the specified time. After all data is migrated, the migration job ends.

      If you change the data at the source data address after the migration job is complete, you can submit another job to migrate all data. In this case, the system migrates only the changed data.

    • Incremental: The system runs a migration job based on the values that you specified for the Migration Interval and Migration Times parameters.
      • You must configure the Start Time Point of File parameter. During the first migration, the system migrates all data whose last modification time is later than the specified time. After the first migration is complete, incremental migration is performed based on the value that you specified for the Migration Interval parameter. For each incremental migration, the system migrates only the data that is created or modified after the previous migration starts and before the current migration starts.
      • If you set the Migration Times parameter to N, full migration is performed once and then incremental migration is performed (N - 1) times.

        For example, the current time is 08:00, March 10, 2019 and you set the Migration Interval parameter to 1, the Migration Times parameter to 5, and the Start Time Point of File parameter to 08:00, March 5, 2019. When the first migration starts, the system migrates all files whose last modification time is between 08:00, March 5, 2019 and 08:00, March 10, 2019. In this example, the first migration requires one hour to complete and the second migration starts at 10:00, March 10, 2019. The duration from 08:00 to 10:00 consists of one hour for the first migration process and one hour for the migration interval. During the second migration, the system migrates the files whose last modification time is between 08:00, March 10, 2019 and 10:00, March 10, 2019. The migration job consists of one full migration and four incremental migrations.

    Important Before full or incremental migration starts, the system compares the files at the source data address with the files at the destination data address. If a source file has the same name as a destination file, the destination file is overwritten when one of the following conditions is met:
    • The Content-Type values of the source file and the destination file are different.
    • The last modification time of the source file is later than that of the destination file.
    • The size of the source file is different from that of the destination file.
    Start Time Point of FileYesSpecify a time to filter data to be migrated. Data will be migrated if it is created or modified after the specified time. Valid values:
    • All: migrates all files.
    • Assign: migrates only the files that are created or modified after the specified time.

      For example, if you set the time to 08:00:00, November 1, 2018, only the files that are created or modified after 08:00:00, November 1, 2018 are migrated.

    Migration IntervalThis parameter is required if you set the Migration Type parameter to Incremental.The default value is 1, and the maximum value is 24. Unit: hours.
    Migration TimesThis parameter is required if you set the Migration Type parameter to Incremental.The default value is 1, and the maximum value is 30.
    File Overwrite MethodYesThe method that is used to process the source file whose name is the same as that of a destination file. Valid values:
    • LastModified: If a source file has the same name as a destination file, the system compares the last modification time that is indicated by the LastModified property of the files.
      • If the last modification time of the source file is later than that of the destination file, the source file is migrated and the destination file is overwritten.
      • If the last modification time of the source file is earlier than that of the destination file, the source file is not migrated and the destination file is retained.
      • If the two files have the same last modification time, the system checks the values of the Size and Content-Type properties of the files.

        If the two files have the same Size and Content-Type values, the source file is not migrated. If the two files have different values of at least one property, the source file is migrated and the destination file is overwritten.

    • Condition: If a source file has the same name as a destination file, the system compares the values of the LastModified, Size, and Content-Type properties of the two files.
      • If all property values of the two files are the same, the source file is not migrated.
      • If the two files have different values of at least one property, the source file is migrated and the destination file is overwritten.
    • All: If a source file has the same name as a destination file, the system performs no comparison and directly overwrites the destination file with the source file.
    • No: If a source file has the same name as a destination file, the system performs no comparison and directly skips the source file during the migration.
  5. On the Performance tab, navigate to the Data Prediction section and specify the Data Size and File Count parameters.
    Note To ensure a successful migration, estimate the amount of data to be migrated as accurately as possible. For more information, see Estimate the amount of data to be migrated.
  6. Optional. On the Performance tab, navigate to the Flow Control section, specify the Time Range and Max Flow parameters, and then click Add.
    Note To ensure business continuity, we recommend that you specify the Time Range and Max Flow parameters based on the workload peaks and troughs.
  7. Click Create and wait until the migration job is completed.