This topic describes the usage notes and procedure for data migration.

Usage notes

When you perform an online migration job, take note of the following items:
  • A migration job occupies the network bandwidths of the source and destination data addresses. To ensure business continuity, we recommend that you configure a rate limit for your migration job or run your migration job during off-peak hours.
  • Before an online migration job starts, the system checks the files at the source data address and the destination data address. If the source and destination have files with the same name, and the last modified time of the file in the source is later than that of the file in the destination, the file in the destination is overwritten. If both files are important and contain different information, we recommend that you change the name of one file or back up the destination file.
  • Data Online Migration allows you to migrate only the data of a single bucket per job. You cannot migrate all data that belongs to your account in a single job.

Step 1: Create a source data address

  1. Log on to the Alibaba Cloud Data Transport console.
  2. In the left-side navigation pane, choose Data Online Migration > Data Address. On the page that appears, click Create Data Address.
  3. In the Create Data Address panel, configure the parameters and click OK. The following table describes the parameters.
    Parameter Required Description
    Data types Yes Select Google Storage.
    Data Name Yes Enter a name. The name must be 3 to 63 characters in length. The name cannot contain special characters, except for hyphens (-) and underscores (_).
    Bucket Yes Enter a name for the bucket in which the data you want to migrate is stored.
    Prefix Yes
    • Migrate All Data: All data in the bucket is migrated.

      If you select this option, you do not need to specify a prefix.

    • Migrate Partial Data: Only the files in the directory whose name contains the specified prefix are migrated. A prefix must start and end with a forward slash (/). For example, you can specify /data/to/oss/ or /docs/ as a prefix.
    Key File Yes Upload the JSON file that you obtained during preparation.
  4. Optional:If the system prompts that the feature is in public preview, click Application to apply to use the feature.
  5. Optional:Enter the required information and submit the application. After the application is approved, the system notifies you by text message.

Step 2: Create a destination data address

  1. In the left-side navigation pane, choose Data Online Migration > Data Address. On the page that appears, click Create Data Address.
  2. In the Create Data Address panel, configure the parameters and click OK. The following table describes the parameters.
    Parameter Required Description
    Data types Yes Select OSS.
    Data Name Yes Enter a name. The name must be 3 to 63 characters in length. The name cannot contain special characters, except for hyphens (-) and underscores (_).
    Data Region Yes Select the region to which the destination data address belongs.
    Enable Transfer Acceleration No If you want to use Data Online Migration together with the transfer acceleration feature of Object Storage Service (OSS), you must enable the transfer acceleration feature for buckets. The transfer acceleration feature takes effect within 30 minutes after it is enabled. After you enable the feature, wait for 30 minutes before you create a migration job.
    Notice If you use the transfer acceleration feature to access your OSS buckets, you are charged additional transfer acceleration fees. For more information, see Transfer acceleration.
    OSS Endpoint Yes
    Select an endpoint based on the region in which your data is stored. For more information, see Regions and endpoints.
    • http://oss-cn-endpoint.aliyuncs.com: indicates that you use an HTTP-based endpoint to access Object Storage Service (OSS) over the Internet.
    • https://oss-cn-endpoint.aliyuncs.com: indicates that you use an HTTPS-based endpoint to access OSS over the Internet.
    Note If you migrate third-party data to OSS, you must access OSS from an Internet endpoint.
    Access Key Id Yes Specify the AccessKey pair that you want to use to migrate data. For more information, see Prepare for a migration.
    Access Key Secret Yes
    OSS Bucket Yes Select a bucket to store the migrated data.
    OSS Prefix No An OSS prefix cannot start with a forward slash (/) and must end with a forward slash (/). For example, you can specify data/to/oss/ as an OSS prefix. If you do not configure this parameter, the root directory (/) is used.
    Notice If the name of a source file starts with a forward slash (/), you must specify an OSS prefix when you specify the destination data address. If no OSS prefix is specified, the migration job fails. For example, if the name of a file that you want to migrate is /test/test.png, you must specify an OSS prefix. Example: oss/. After the migration job is complete, the name of the OSS file changes from /test/test.png to oss//test/test.png.

Step 3: Create a migration job

  1. Choose Data Online Migration > Migration Jobs and click Create Job.
  2. In the Create Job panel, read the terms of the migration service and select I understand the above terms and conditions, and apply for opening data migration service. Then, click Next.
    In the Fee Reminder dialog box, click Yes, Go Ahead. OSS request fee
  3. In the Job Config step, configure the following parameters and click Next.
    Parameter Required Description
    Job Name Yes Enter a job name. The name must be 3 to 63 characters in length and can contain lowercase letters, digits, and hyphens (-), but cannot start or end with a hyphen (-).
    Source Data Address Yes Select the source data address that you created.
    Destination Data Address Yes Select the destination data address that you created.
    Notice If the source and destination data addresses are located in different countries, make sure that you have the permissions to create cross-country migration jobs. To request the permissions, submit a ticket. You must declare that your business is legitimate, the data transmission process conforms to local rules and regulations, and your data does not contain illegal information.
    Specified Directory No Specify the directories whose files and subdirectories are included or excluded during migration.
    • Do not filter: All data from the source data address is migrated.
    • Exclude: The files and subdirectories in the specified directories are not migrated.
    • Contain: Only the files and subdirectories in the specified directories are migrated.
    Note
    • You must configure a whitelist before you set the Specified Directory parameter. For more information, see Advanced features.
    • A directory name cannot start with a forward slash (/) or a backslash (\), and cannot contain double slashes (//), double periods (..), or double quotation marks ("). The character string that consists of all the specified directory names cannot exceed 10 KB in length.
    • A directory name must end with a forward slash (/). For example, you can specify docs/ as the directory name.
    • You can specify a maximum of 20 directories of the Exclude or Contain type.
    Migration Type Yes Select the data migration mode.
    • Full: The system migrates all data from the source data address to the destination data address. You must specify the Start Time Point of File parameter. The system migrates all data whose last modified time is greater than the specified time. After all data is migrated, the migration job ends.

      If the data at the source data address changes after the migration job is complete, you can submit another full migration job. In this case, the system migrates only the data that has changed since the last migration job.

    • Incremental: The system runs migration jobs based on the specified Migration Interval and Migration Times parameters.
      • You must specify the Start Time Point of File parameter. On the first migration job, the system first migrates all data whose last modified time is greater than the specified time. After the first migration job is complete, incremental migration jobs are performed at a regular interval based on the Migration Interval parameter. In each subsequent migration job, the system migrates only the data that is created or modified after the previous migration job starts and before the current migration job starts.
      • If you set Migration Times to N, a full migration job is performed once and then the incremental migration job is performed (N - 1) times.

        For example, assume that the current time is 08:00, March 10, 2019 and you set Migration Interval to 1 hour, Migration Times to 5, and Start Time Point of File to 08:00, March 5, 2019. The system first performs a full migration job of all files whose last modified time is between 08:00, March 5, 2019 and 08:00, March 10, 2019. In this example, the full migration job takes one hour to complete. After the full migration job is complete, the system waits one hour to start the first incremental migration job at 10:00, March 10, 2019. During the first incremental migration job, the system migrates the files whose last modified time is between 08:00, March 10, 2019 and 10:00, March 10, 2019. The system performs one full migration job and four incremental migration jobs in total.

    Notice Before a full or incremental migration job starts, the system compares files at the source data address with those at the destination data address. If a source file has the same name as a destination file, the destination file is overwritten when one of the following conditions is met:
    • The Content-Type values of the source file and the destination file are different.
    • The last modified time of the source file is later than that of the destination file.
    • The size of the source file is different from that of the destination file.
    Start Time Point of File Yes Specify a time to filter data that the system needs to migrate. Data will be migrated if it is created or modified after the specified time.
    • All: All files are migrated.
    • Assign: Only the files that are created or modified after the specified time are migrated.

      For example, if you set the time to 08:00:00, November 1, 2018, the system migrates only the files that are created or modified after 08:00:00, November 1, 2018.

    Migration Interval This parameter is required when you set Migration Type to Incremental. The default value is one hour and the maximum value is 24 hours.
    Migration Times This parameter is required when you set Migration Type to Incremental. The default value is 1 and the maximum value is 30.
    File Overwrite Method Yes The method used to process the source file whose name is the same as that of a destination file. Valid values:
    • LastModified: If a source file has the same name as a destination file, the system compares the last modified time indicated by the LastModified property of the files.
      • If the last modified time of the source file is later than that of the destination file, the source file is migrated and the destination file is overwritten.
      • If the last modified time of the source file is earlier than that of the destination file, the source file is not migrated.
      • If the last modified time of the two files are the same, the system compares the Size and Content-Type property values of the files.

        If the two files have the same property values, the source file is not migrated. If the values of at least one property are different, the source file is migrated and the destination file is overwritten.

    • Condition: If a source file has the same name as a destination file, the system compares the values of the LastModified, Size, and Content-Type properties of the files.
      • If all property values of the two files are the same, the source file is not migrated.
      • If the two files have different values of at least one property, the source file is migrated and the destination file is overwritten.
    • All: If a source file has the same name as a destination file, the system performs no comparison and directly overwrites the destination file with the source file.
    • No: If a source file has the same name as a destination file, the system performs no comparison and directly skips the source file during the migration.
  4. In the Data Prediction section of the Performance step, configure the Data Size and File Count parameters.
    Note To ensure a successful migration job, we recommend that you estimate the volume of data that you want to migrate as accurately as possible. For more information, see Estimate the volume of data that you want to migrate.
  5. Optional:In the Flow Control section of the Performance step, configure the Time Range and Max Flow(MB/s) parameters and click Add.
    Note To ensure business continuity, we recommend that you configure the Time Range and Max Flow parameters based on the workload peaks and troughs.
  6. Click Create and wait until the migration job is complete.