Object Storage Service:Data migration overview

Migration within the same region

When the source bucket and the destination bucket reside in the same region, select a migration method based on your scenario:

• ossutil — For small to medium amounts of data in buckets that belong to the same account, use the ossutil command line tool to copy objects between buckets. ossutil supports batch object copy and resumable transfer.

• Same-region replication — For large amounts of data or for migration between buckets that belong to different accounts, use same-region replication. Data is not downloaded to an intermediate location and no extra network transfer is required. When objects are added, modified, or deleted in the source bucket, the changes are automatically synchronized to the destination bucket. This method suits scenarios in which data is centralized and shared across teams or subsidiaries.

Migration across regions

When the source bucket and the destination bucket reside in different regions, for example when you migrate data from China (Hangzhou) to China (Beijing), use cross-region replication. Data is transferred over the internal network of Alibaba Cloud. Replication is continuous: object additions, modifications, and deletions in the source bucket are synchronized to the destination bucket automatically. This method suits multi-site collaboration and real-time backup scenarios.

Storage region changes for resource plans

If you need to switch storage regions due to regional restrictions on resource plans, migrate data to the bucket in the destination region by using cross-region replication or ossutil. Cross-region replication is configured on the source bucket. After the migration is complete, purchase a resource plan in the destination region, because a purchased resource plan cannot be transferred to a different region.

Migration from a decommissioned region

When a bucket resides in a region that is being decommissioned, such as China (Nanjing - Local Region), use cross-region replication to migrate data to a bucket in another available region. Cross-region replication is billed based on replication traffic. For CRR across accounts, the account to which the source bucket belongs bears the cost.

Cross-region replication does not automatically delete data from the source bucket. After you confirm that the migration is complete, manually delete the source data based on your business requirements. If the account balance is insufficient during data transfer, replication tasks may be interrupted or fail. Make sure that your account has a sufficient balance before you start the migration.

Third-party cloud storage services

To migrate data that is stored with another cloud service provider to Alibaba Cloud OSS, use Data Online Migration. Storage services such as AWS S3, Tencent Cloud COS, Huawei Cloud OBS, Volcengine TOS, Google Cloud GCS, and Microsoft Azure Blob are supported. You do not need to set up a migration environment. You can submit migration tasks online and monitor the migration progress at any time. Migration tasks support file filtering: you can filter the files to migrate by file name, with RE2 regular expressions, and by last modified time. For the procedure that matches your business scenario, see the Data Online Migration tutorial.

Self-managed S3-compatible object storage

Data Online Migration also supports self-managed object storage services that are compatible with the S3 protocol, such as OpenStack Swift with the S3-compatible API enabled.

Small amounts of data

For less than 5 GB of data, upload the data directly in the OSS console. This method requires no additional tools or complex configurations, and you can upload files in the browser. It suits temporary file uploads, test data migration, infrequent migration tasks, and scattered files.

Medium amounts of data

To migrate a medium amount of data, select one of the following methods based on your requirements:

• ossutil — To transfer regular business data, log files, or backup data, use the ossutil command line tool to upload local files to OSS. ossutil supports batch uploads, resumable transfer, and concurrent acceleration.

• Data Online Migration — If your on-premises network is complex, or if you manage migration tasks centrally in the cloud, use Data Online Migration to migrate data from LocalFS to OSS. This managed migration solution supports task creation, monitoring, and management, which reduces operations complexity. It suits enterprise customers that need centralized control over the migration process.

Large amounts of data

To migrate a large amount of data, select one of the following methods based on your requirements:

• Data Online Migration — Use Data Online Migration when the data can be transferred over the Internet. You do not need to set up a migration environment, and you can submit migration tasks online and monitor the migration progress in real time. This method suits business scenarios that require flexible scheduling.

• Data Transport — If your data transfer is limited by Internet bandwidth, or if you migrate data from an on-premises data center, use Data Transport. Data Transport is designed for migration of TB-scale to PB-scale data. Data is collected and transferred on physical devices, which removes the Internet bandwidth bottleneck. This method suits scenarios such as migrating an entire data center to the cloud, enterprise archive files, and historical image data.

HDFS data to OSS

For large amounts of data in a Hadoop Distributed File System (HDFS), use Jindo DistCp to migrate HDFS data to OSS. Jindo DistCp is a distributed file copy tool that is based on MapReduce and transfers files within a large cluster or between clusters. Jindo DistCp uses the file and directory lists as the input of a MapReduce job and splits the job into parallel tasks. Each task copies part of the data in the source list. Jindo DistCp supports fault tolerance, resumable transfer, and error recovery to maintain stability and data integrity in TB-scale or even PB-scale migration. This tool suits scenarios such as big data computing and data lake construction.

Data exchange between OSS and AnalyticDB for PostgreSQL

If your data is stored as OSS foreign tables (gpossext) and you need to import and export data between OSS and a data warehouse, use AnalyticDB for PostgreSQL: export data to OSS in parallel. The gpossext feature imports data from OSS to AnalyticDB for PostgreSQL in parallel, or exports data to OSS in parallel. Based on a distributed architecture, the migration provides high concurrency and high throughput capacity, which shortens the transfer time and maintains data security and data integrity. This method suits scenarios such as large-scale data analytics, historical data archiving, and cross-system data exchange.

Traditional HDFS clusters to OSS-HDFS

To migrate data from a traditional HDFS cluster, use Jindo DistCp to migrate HDFS data to OSS-HDFS. This tool distributes large numbers of files within a cluster or across clusters and supports automatic error detection, retries, and task recovery, which improves the stability of migration tasks. Jindo DistCp uses the file and directory lists as the input of a MapReduce job, and each task copies part of the files. This approach suits migration of massive amounts of data in batches.

Data between OSS-HDFS buckets

To migrate data from one OSS-HDFS bucket to another, use Jindo DistCp to migrate data between OSS-HDFS buckets. This method suits requirements such as adjusting data partitions, optimizing storage resources, and scheduling data across regions, and it keeps business data consistent and highly available.

Semi-managed JindoFS clusters to OSS-HDFS

If you use a semi-managed JindoFS cluster, use JindoDistJob to migrate data from a semi-managed JindoFS cluster to OSS-HDFS. This tool supports full migration and incremental migration, and it switches to the JindoFS service-based solution without migrating data blocks. The migration does not affect your business, which suits scenarios in which you need to switch the storage architecture quickly.

Hive tables and partition data to OSS-HDFS

To migrate structured data, use the JindoTable MoveTo command to migrate Hive tables and partition data to OSS-HDFS. After the underlying data is copied, the command automatically updates the metadata so that the tables and partitions are fully migrated to the new path. The command supports filter conditions, migrates a large number of partitions at a time, and uses multiple data validation mechanisms to maintain data integrity and data security. This command suits migration of large structured datasets.