This topic describes how to migrate data of an Internet-based application from a third-party storage service to Alibaba Cloud Object Storage Service (OSS) by using Data Online Migration together with mirroring-based back-to-origin.
Background information
Enterprise A provides an Internet-based application that can be used to modify media files such as images and videos. The main components of the application are deployed in a cloud-based architecture provided by Enterprise B. Enterprise A uses a cloud-based storage service provided by Enterprise B to store data. The media file editing application contains approximately 100,000,000 existing files and the total size of data is approximately 320 TB. The size of data increases by 20 GB every day. The business scenario in which the application is used requires a maximum bandwidth of 50 Mbit/s. OSS provides a bandwidth of 250 Mbit/s for data writes and reads.
To grow their business, Enterprise A wants to migrate data to OSS. Existing data and incremental data must be migrated to OSS. In view of the large amount of historical data, to ensure that the data migration does not interrupt business operations, the following requirements must be met:
During the migration, the customers of Enterprise A can read data.
After the migration is complete, all data of the application is migrated to OSS and the application can run as expected without interruptions.
Migration solution
The following migration solution is created based on the requirements and background information of Enterprise A:
Use Data Online Migration to migrate the customer's historical data from the original cloud service to Alibaba Cloud OSS. The customer's business operations remain unchanged before the migration is complete.
After the historical data migration is complete, configure mirroring-based back-to-origin from OSS to the original cloud service. This ensures that incremental data not yet migrated to OSS can be accessed through OSS.
The customer switches their business to OSS. At this point, all read and write operations are performed in OSS.
After the business switch is complete, no more incremental data is generated in the original cloud service. Use Data Online Migration to migrate the customer's incremental data to OSS.
After the migration is complete, you must check and make sure that all data is migrated to OSS, and then delete the data from the source.
Step 1: Migrate existing data
Create a migration job to migrate existing data.
a. Create an OSS bucket to store the migrated data.
b. In the Data Online Migration console, create a migration job. For more information, see Migrate data from BOS to OSS.
ImportantSet Overwrite Method to Overwrite based on the last modification time. When you restart the migration, only incremental data are migrated.
After the migration is complete, you can view the migration report and compare data in the source and destination addresses to check whether all data is migrated.
Step 2: Create mirroring-based back-to-origin rules
The job requires approximately 25 days to migrate existing data. During the migration, incremental data is generated and stored in the source data address. To ensure business continuity and a seamless business switchover, you must create mirroring-based back-to-origin rules. After you configure the rules, if the files that are requested by users do not exist in the bucket, OSS retrieves these files from the origin, stores the files in the bucket, and returns the files to the users.
Log on to the OSS console.
In the left-side navigation pane, click Buckets.
On the Buckets page, click the name of the bucket for which you want to create a mirroring-based back-to-origin rule.
In the left-side navigation pane, choose .
On the Mirroring-based Back-to-origin page, click Create Rule. In the Create Rule panel, configure the parameters.
Method: Select Mirroring.
Condition: By default, HTTP Status Code 404 is selected. You can specify a prefix and a suffix for the names of files.
Origin URL: Enter the URL of the source data address.
For more information about how to configure the parameters, see Overview.
NoteYou can create up to five mirroring-based back-to-origin rules for a bucket. The created rules take effect at the same time. If the data is migrated from multiple source data addresses, you can create multiple mirroring-based back-to-origin rules and specify different prefixes for the files.
Click OK.
Step 3: Change the data store of the application to OSS
Modify the configurations of the servers to change the URL to which the servers connect to write data to and read data from the bucket. Incremental data is not generated in the source data address.
Step 4: Migrate incremental data
During data migration, the application generates approximately 100,000 files that contain approximately 500 GB of data. These files must be migrated to the bucket.
Restart the migration job. Scan full data in the source data address and migrate incremental files to the bucket based on the last modified time.
After the migration is complete, you can view the migration report and compare data in the source and destination addresses to check whether all data is migrated.