This topic describes how to migrate the data of a web-based service provider from a cloud service to Alibaba Cloud Object Storage Service (OSS).

Background information

Enterprise A is a web-based service provider that offers services for editing media files, such as images and videos. The main business applications are deployed in a cloud-based architecture provided by Enterprise B. The data that is stored on the servers of Enterprise B includes 100,000,000 files and has a total size of approximately 320 TB. The size of the data increases by 20 GB each day. The bandwidth for the storage service of Enterprise B 250 MB/s. The bandwidth for OSS is also 250 MB/s. The business applications require a maximum bandwidth of 50 MB/s.

To accelerate business growth, Enterprise A wants to move the business applications to OSS. Both the existing data and incremental data need to be migrated to OSS. To ensure the successful migration of large amounts of historical data and business continuity, the following requirements must be met:
  • During the migration, users can read data as normal.
  • After the migration, the data is complete and the business applications can run as expected without service interruptions.

Migration solutions

The following data migration solution is based on the customer requirements and background information:
  1. Use Data Online Migration to migrate the existing data of Enterprise A from a cloud service provided by Enterprise B to OSS. Make sure that the customer updates no data during the entire migration process.
  2. After the existing data is migrated, create back-to-origin rules in OSS for users to access the incremental data that is migrated.
  3. Switch the business applications to OSS.
  4. After the business applications are switched to OSS, use Data Online Migration to migrate incremental data to OSS.
  5. After all data is migrated and validated, delete the data from the original source.

Step 1: Migrate the existing data

  1. Create an OSS bucket to store the migrated data. For more information, see Create buckets.
  2. Obtain the AccessKey pairs that are used to migrate data:
    • To obtain the AccessKey pair provided by Enterprise B, log on to the cloud service console and view the AccessKey pair.
    • Obtain the AccessKey pair of the OSS Resource Access Management (RAM) user. For more information, see Create a RAM user and grant permissions to the RAM user.
  3. Create data addresses and a full migration job. For more information, see the topics about data migration tutorials in the documentation of Data Online Migration.
    In the Job Config step, configure the migration job. The following figure provides an example of the configurations. Job configuration
    In the Performance step, configure the performance parameters. The following figure provides an example of the configurations. Performance
  4. After the migration, view the migration report and compare data between the source data address and the destination data address to ensure that all data is migrated.
    Note If a file fails to be migrated, troubleshoot the failure. For information about migration failures, see Common causes of a migration failure and solutions.

Step 2: Create back-to-origin rules

It takes approximately 25 days to migrate the existing data. During the migration process, data at the source data address is continuously growing. To ensure business continuity and a seamless business switchover, create back-to-origin rules. If a file that you request does not exist in OSS, OSS fetches the file from its source data address and returns it to you based on the back-to-origin rules.

  1. Log on to the OSS console.
  2. In the list of buckets, select the bucket in which the migrated data is stored.
  3. In the left-navigation pane, choose Basic Settings > Back-to-Origin. Then, click Configure.
  4. Click Create Rule. In the Create Rule dialog box, configure the parameters.
    Create a rule.
    • Mode: Select Mirroring.
    • Prerequisite: HTTP Status Code 404 is selected by default. Configure File Name Prefix based on your needs. File Name Prefix can be left empty.
    • Origin URL: Enter the endpoint of the cloud service where the source data resides.
    • For more information about the parameter configuration, see Create back-to-origin rules.
    Note You can create a maximum of five back-to-origin rules. The five rules can be in effect at the same time. For multiple source data addresses, you can create multiple back-to-origin rules. You can set different rules by specifying different values for File Name Prefix so that OSS can fetch various types of data.
  5. Click OK.

Step 3: Switch the business to OSS

Change the data address from which Enterprise A obtains data to an OSS data address.

Step 4: Migrate the incremental data

During the migration of the existing data, approximately 100,000 files with a total size of about 500 GB are generated at the source data address. You must migrate these incremental data to OSS.

  1. Create an incremental migration job based on the instructions in Step 1.

    In the Job Config step, configure the migration job. The following figure provides an example of the configurations.

    Job configurationIn the Performance step, configure the performance parameters. The following figure provides an example of the configurations. Performance
  2. Click Create to migrate data.
  3. After the migration, view the migration report and compare data between the source data address and the destination data address to ensure that all data is migrated.
    Note If a file fails to be migrated, troubleshoot the failure. For information about migration failures, see Common causes of a migration failure and solutions.

Step 5: Delete the data at the source data address

You can create a lifecycle rule to delete the files at the source data address one day after all data is migrated. This way, you are no longer charged for storing the data at the source data address one day after the data is migrated.