This topic describes how to migrate data of an Internet enterprise from a cloud service to Alibaba Cloud Object Storage Service (OSS).

Background information

Enterprise A is an Internet service provider that deploys the main business applications in a cloud service provided by Cloud Service Provider B. The main business applications of Enterprise A provide online editing services for media files, such as images and videos. The existing data that is stored on the servers of Cloud Service Provider B includes 100,000,000 files and has a total size of about 320 TB with a daily increase of 20 GB. The bandwidth for both the storage service of Cloud Service Provider B and OSS is 250 MB/s. The business applications require a maximum bandwidth of 50 MB/s.

Enterprise A wants to switch over the businesses to OSS based on the development needs. When you switch over businesses between two data stores, you must migrate existing data and incremental data to OSS. To ensure a successful migration of large amounts of historical data and business continuity, the following needs must be met:
  • During migration, you must ensure business continuity and avoid the impacts on normal data access from users.
  • After the migration job is complete, you must check data integrity to ensure a seamless switchover of the businesses to OSS.

Migration solution

Based on the background information, you can migrate data as follows:
  1. With Data Transport, you can migrate existing data from a cloud service to OSS. Before a migration job is complete, ensure that no updates occur on the customer side.
  2. After existing data is migrated, you can create back-to-origin rules in OSS for users to access the incremental data that has not been migrated.
  3. Switch over businesses to OSS.
  4. After the business switchover is complete, you can migrate incremental data to OSS by using Data Transport.
  5. After all data is migrated and validated, delete the data at the source data address.

Step 1: Migrate existing data

  1. Create an OSS bucket to store migrated data. For more information, see Create a bucket.
  2. Obtain the AccessKey pairs that are used to migrate data as follows:
    • To obtain the AccessKey pair provided by Cloud Service Provider B to access the storage service, log on to the cloud service console and view the AccessKey pair.
    • You must also obtain the AccessKey pair of a RAM user account that is used to access OSS. For more information, see Create and grant permissions to a RAM user.
  3. Create data addresses and a full migration job. For more information, see the corresponding topics in the Data Transport documentation.
    Configure the required parameters on the Job Config tab as follows:Job configurations
    The following figure shows the parameters that you can specify in the Performance step.Performance optimization
  4. To ensure that all data is migrated after migration, you need to view the migration report and compare data at both the source data address and the destination data address.
    Note For more information about how to troubleshoot migration issues, see Common causes of a migration failure and solutions.

Step 2: Create back-to-origin rules

The migration of the existing data takes about 25 days. During the migration process, data is continuously growing at the source data address. To ensure business continuity and a seamless switchover, you need to create back-to-origin rules. When files that are requested by end users do not exist in OSS, OSS fetches these files from the source data address and returns them to end users.

  1. Log on to the OSS console.
  2. In the list of buckets, select the bucket where migrated data is located.
  3. Select Basic Settings and click Configure in the Back-to-Origin section.
  4. Click Create Rule. In the Create Rule dialog box, configure the required parameters.
    Create a rule.
    • Mode: Select Mirroring.
    • Prerequisite: HTTP Status Code 404 is selected by default. You can configure the File Name Prefix as needed.
    • Origin URL: Enter the endpoint of an Alibaba Cloud service.
    • For more information about configuration parameters, see Create back-to-origin rules.
    Note You can create a maximum of five back-to-origin rules. The five rules take effect at the same time. For multiple source data addresses, you can create multiple back-to-origin rules. You can enable OSS to fetch various types of data by setting different values for the File Name Prefix.
  5. Click OK to complete the configuration.

Step 3: Switch over businesses to OSS

Change the data address where the business applications obtain data to OSS.

Step 4: Migrate incremental data

During the migration of existing data, about 100,000 files with a total size of about 500 GB are generated. You must migrate these incremental data to OSS.

  1. Create an incremental migration job based on the instructions described in Step 1.

    Configure the required parameters on the Job Config tab as follows:

    Job configurationsThe following figure shows the parameters that you can specify in the Performance step.Performance optimization
  2. Click Create to create a migration job.
  3. To ensure that all data is migrated after migration, you need to view the migration report and compare data at both the source data address and the destination data address.
    Note For more information about how to troubleshoot migration issues, see Common causes of a migration failure and solutions.

Step 5: Delete data at the source data address

After a migration job is complete, you can create a lifecycle rule for files at the source data address to avoid extra charges for storage. This rule sets the validity period for files to one day. All data at the source data address is deleted on the expiration date.