Migration Hub:Solution for migrating object storage from other cloud services to Alibaba Cloud OSS

1. Scenarios

This solution applies to scenarios where you need to migrate data from the object storage of other cloud services to Alibaba Cloud Object Storage Service (OSS). It covers use cases such as enterprise-level data migration, cloud-native architecture transformation, and hybrid cloud deployment.

1.1 Applicable cloud service provider products

1.2 Applicable migration types

• Full data migration: Suitable for initial migrations or small-scale data scenarios that require a one-time migration of all source data.

• Incremental data migration: Suitable for scenarios with continuous data updates. This type of migration uses real-time synchronization to ensure data consistency.

1.3 Applicable storage types

• Standard: Suitable for frequently accessed business data.

• Infrequent Access: Suitable for data that is accessed periodically, at a lower cost.

• Archive Storage: Suitable for long-term cold data storage. This storage type supports on-demand restoration.

2. Introduction to migration tools

1. Alibaba Cloud Data Online Migration

Alibaba Cloud Data Online Migration is a data tunnel service for Alibaba Cloud storage products. You can use Data Online Migration to easily migrate third-party data to Alibaba Cloud OSS or perform flexible data migrations between OSS buckets.

To use Data Online Migration, you can enter the source data address and destination OSS address in the console to create a migration task. After you start the migration, you can manage the task and view information, such as migration progress and traffic, in the console.

2. Alibaba Cloud CMH Cloud Migration Hub

Data migration is a common scenario, and data accuracy directly affects business operations. Therefore, ensuring data consistency before and after migration is critical. Cloud Migration Hub (CMH) provides multiple validation methods for various data sources, such as different database and object storage types, to ensure data consistency between the source and destination. This topic describes the data validation procedure to help you quickly learn how to use the data validation feature.

3. Migration plan

1. Overall migration flow

Storage creation:

Create a bucket in Alibaba Cloud OSS.

1. Select a region:

   • Factors to consider include latency (proximity to users), compliance (such as data residency requirements), and cost (price differences between regions).

   • Naming convention: Use a unique and compliant name. For example, <business-name>-<environment>-<timestamp>.

2. Select a storage class:

   • Storage classes:

     • Standard: Suitable for frequently accessed data.

     • Infrequent Access (IA): Suitable for cold data at a lower cost.

     • Archive Storage: Suitable for extremely infrequently accessed data at the lowest cost.

   • Versioning: You can enable versioning to prevent data from being overwritten or deleted.

   • Cross-region replication (optional): You can configure cross-region synchronization. For example, the primary bucket is in China (Shanghai) and the backup bucket is in China (Beijing).

3. Bucket policy:

   • You can configure an access policy to restrict access. For example, you can allow access from specific IP addresses or Resource Access Management (RAM) users.

Lifecycle migration:

You can automate the management of storage class transitions and object expiration.

1. Planning policy:

   • You can define rules based on data access frequency. For example, you can transition objects to the IA storage class after 7 days and to Archive Storage after 30 days.

   • Deletion policy: For example, you can delete data after 180 days.

2. Create a lifecycle rule:

   • You can configure rules in the OSS console or using an API. You can configure multiple rules.

   • Example rules:

     • Rule 1: Transition objects to the IA storage class 7 days after they are created.

     • Rule 2: Delete objects 180 days after they are created.

3. Validation and monitoring:

   • You can use Simple Log Service for OSS to monitor the execution status of rules.

   • Periodically check whether the storage class of files in the bucket meets expectations.

Security policy migration:

Ensure that the permissions and security policies in OSS are consistent with the source system after migration.

1. Permission migration:

   • RAM roles and policies: You can export the RAM permission policies from the source system, import them into Alibaba Cloud, and attach them to the corresponding users or roles.

   • Bucket policy: You can export the bucket policy from the source bucket, adapt it to the OSS syntax, and then reconfigure it.

   • ACL migration: If the source system uses access control lists (ACLs), you can convert them to a bucket policy or a RAM policy.

2. Encryption migration:

   • If the source data is encrypted, you can use Key Management Service (KMS) for OSS to migrate the keys or re-encrypt the data.

Data migration:

• You can use Alibaba Cloud Data Online Migration to migrate data.

Data validation:

• You can use Alibaba Cloud CMH to validate data.

2. Data migration plan

2.1 Use Alibaba Cloud Data Online Migration for data migration

Prerequisites:

• If your source data is in the Archive storage class, you must restore the data before you create a migration task. After the data is restored, you can create the migration task. Note the following items during the restoration:

  • Make sure to complete the restore operation before you create the source data address and the migration task.

  • Ensure that the restoration duration is sufficient for the total amount of data to be migrated. This prevents the data from being archived again during the migration.

  • A data retrieval fee is charged for the restore operation. Some restoration services may have high fees. For information about specific billing rules, consult your source storage service provider.

Notes:

• Data Online Migration uses the public standard interfaces of the source storage service provider to access source data. Its behavior depends on the specific implementation of the source storage service provider.

• Data Online Migration consumes resources at both the source and destination, which may affect normal business operations. If your business is critical, you should evaluate the impact and set a rate limit for the migration task, or start the migration task during off-peak hours.

• Before the migration, Data Online Migration checks the files at the source and destination. However, if files with the same name exist at both the source and destination and you set the overwrite mode to Overwrite in the migration task, the files at the destination are directly overwritten during the migration. If the content of the two files is different, you must rename the files or create backups.

• Data Online Migration retains the last modified time property of the source files. If a lifecycle rule is set for the destination bucket and the migrated files fall within the time range when the rule takes effect, the files may be deleted or transitioned to the specified storage class when the rule takes effect.

Limits:

• You can migrate data from only one bucket at a time. You cannot migrate data from an entire account in a single task.

• For data migrated from AWS S3:

  • Supported properties: x-amz-meta-*, LastModifyTime, Content-Type, Cache-Control, Content-Encoding, Content-Disposition, Content-Language, and Expires.

  • Unsupported properties: StorageClass, Acl, server-side encryption, and Tagging.

• For data migrated from Tencent Cloud COS:

  • Supported properties: x-cos-meta-*, LastModifyTime, Content-Type, Cache-Control, Content-Encoding, Content-Disposition, Content-Language, and Expires.

  • Unsupported properties: StorageClass, Acl, server-side encryption, and Tagging.

• For data migrated from Huawei Cloud OBS:

  • Supported properties: x-obs-meta-*, LastModifyTime, Content-Type, Cache-Control, Content-Encoding, Content-Language, and Expires.

  • Unsupported properties: StorageClass, Content-Disposition, ACL, server-side encryption, and Tagging.

• Huawei OBS parallel file system is not supported.

• For data migrated from Azure Blob:

  • Supported properties: x-ms-meta-*, LastModifyTime, Content-Type, Cache-Control, Content-Encoding, Content-Disposition, Content-Language, and Expires.

  • Unsupported properties: StorageClass, ACL, server-side encryption, and Tagging.

• For data migrated from Google Cloud GCP:

  • Supported properties: x-amz-meta-*, LastModifyTime, Content-Type, Cache-Control, Content-Encoding, Content-Disposition, and Content-Language.

  • Unsupported properties: Expires, StorageClass, ACL, server-side encryption, and Tagging.

3. Data validation plan

3.1 Use Alibaba Cloud Migration Hub (CMH) for data validation

Validation principle:

CMH uses a lightweight data validation algorithm to ensure validation accuracy while reducing resource consumption. This provides reliable quality assurance for the data migration of critical business systems and helps enterprises achieve smooth and secure data migration.

Notes:

If you perform a validation operation while a data migration is in progress, such as full migration or incremental synchronization, the source and destination data may continuously change. This can lead to the following issues:

• Validation result bias

• Incorrect data consistency judgment

• Interference with the migration process

4. Data migration procedure

1. Use Alibaba Cloud Data Online Migration for data migration

The data migration procedure includes the following steps. The actual steps in the Data Online Migration console prevail.

Step 1: Create a source data address

Step 2: Create a destination data address

Step 3: Create a migration task

Step 4: Modify the execution frequency

For more information about specific procedures and parameters, see the following tutorials:

AWS S3 migration tutorial

Tencent Cloud COS migration tutorial

Huawei Cloud OBS object storage and parallel file system migration tutorial

Azure Blob migration tutorial

Google Cloud Storage migration tutorial

5. Data validation procedure

1. Use Alibaba Cloud Migration Hub (CMH) for data validation

The data validation feature involves the following steps:

Step 1: Install the agent.

During data validation, data from the source and destination must be detected and compared. Due to data sensitivity, CMH data validation requires you to provide a server. An Alibaba Cloud ECS instance is recommended. Make sure that this server can connect to both the source and destination data sources. Then, you can install a data validation agent on the server for subsequent validation.

Step 2: Configure the data source and validation plan.

In actual scenarios, you may not want to validate all data in a data source. Therefore, you need to customize the validation scope and method. After the configuration is complete, you can select the appropriate agent for validation.

Step 3: Start the validation.

After the validation task is created, you can send a validation instruction to the agent that corresponds to the task on the page. After the agent completes the validation, it automatically reports the validation result. You do not need to perform any other operations. You can just wait for the validation to complete.

Step 4:

After the agent returns the result, you can view the validation result and details in the CMH console.

These steps provide a GUI-based method to perform consistency checks between the source and destination data, ensuring data consistency and a smooth cutover.

For more information about the procedure and parameters, see Data validation.

6. Overall migration notes

• Incremental synchronization latency: When a bucket contains many files, such as tens of millions or more, a full scan for incremental synchronization can take a long time. Avoid performing catch-up incremental synchronization during the cutover. In this case, you can use other incremental synchronization methods. For example:

  • For incremental synchronization, you can use an event-driven method that subscribes to source changes and synchronizes them to the destination in real time.

  • If data cannot be retrieved in real time due to latency during incremental synchronization, you can configure mirroring-based back-to-origin as a fallback.

• SDK compatibility issues: When you migrate object storage, you may need to adapt the SDKs depending on the source cloud service. A completely smooth migration cannot be guaranteed. For example, this issue may occur when you migrate data from Qiniu Cloud or other commercial object storage services to Alibaba Cloud.

• Check whether your applications store object access addresses directly in a database or have hardcoded object storage endpoints. If so, you must correct the connection addresses in the database or adapt the code to the new addresses.