edit-icon download-icon

How to migrate data to OSS

Last Updated: Feb 07, 2018

The user wants to migrate historical data to one of their target bucket in OSS. The source data to be migrated may be stored in an OSS bucket, a local storage, a third-party cloud storage (for example, AWS S3, Azure Blob, Tencent Cloud COS, Qiniu, Baidu BOS, UPYUN, and Kingsoft S3), or HTTP.

Methods

  • OssImport
    Use the OssImport tool to migrate the data. This tool can migrate historical data of various data sources to OSS in batches. See OssImport documentation.

  • Image origin retrieval
    This method can seamlessly migrate data from an origin site to OSS. It can be used when services are running on the origin site established by the user or another cloud product, and data needs to be migrated to OSS without stopping the services. For more information about how to migrate new data after migrating historical data using the OssImport tool, see Image origin retrieval and Seamless data migration to OSS.

  • Cross-region replication
    This method can replicate data from Bucket A to Bucket B of the same user account. For more information, see Set cross-region replication. Set the historical data synchronization function when using this method. In addition, if you do not want to synchronize the deletion operations in Bucket A to Bucket B, set the synchronization policy to “write synchronization”.

  • Lightning cube
    This method can migrate a large amount (TB or PB level) of offline data in a data center to a cloud. See Lightning Cube for more information.

  • OSS API or SDK
    Users with high development capabilities can use OSS APIs or SDK code to migrate data. For more information, see Copy Object and Upload Part Copy.

Scenario 1: migration of non-OSS data to OSS

In this scenario, the source data is to be migrated to an OSS bucket, but this data is stored in a non-OSS storage, such as a local storage, HTTP, or a third-party cloud storage (for example, AWS S3 and Azure Blob).

The methods are used as follows:

  • OssImport
    Applicable to batch historical data migration to OSS, especially when the data volume is large. The distributed version of OssImport can be used for large-scale data migration.

  • Image original retrieval
    Applicable to in-service hot data migration from an origin site to OSS. This method can meet the requirements for data migration of data and redirection of specific requests.

  • Set up cross-region replication
    Not applicable.

  • Lightning Cube

    • Any demand to migrate TBs or PBs of data from local data centers to the cloud
    • Any demand for a professional solution that migrates data from local data centers to the cloud in a faster, safer, more efficient, and more cost-effective manner
  • OSS APIs/SDKs Copy Object and Upload Part Copy

    • Source data is stored locally: Use OSS APIs/SDKs for code development.
    • If the source data is stored in HTTP or a third-party cloud storage, you must write code to download the data from the data source, and then write code, by using OSS APIs/SDKs, to upload the data to OSS.

Scenario 2: data migration between OSSs

In this scenario, the data is to be migrated from a source OSS bucket to a target OSS bucket. The migration occurs only between two OSS buckets.

The methods are used as follows:

  • OssImport
    Applicable for any situation, including data migration between different directories of a bucket or between different buckets (including buckets of the same account/region or of different accounts/regions). OssImport is particularly suitable for large-scale data migration, for which the distributed version is recommended.

  • OSS Cross-region Replication

    • The source and target buckets belong to the same user but are in different regions.
    • The source and target buckets are of one of the following storage types:
      • Standard for both the source and target buckets.
      • Standard for the source bucket and Infrequent Access for the target bucket.
      • Infrequent Access for the source bucket and Standard for the target bucket.
      • Infrequent Access for both the source and target buckets.

        Note: If you do not want to synchronize any changes (additions, updates, and deletions) to the source bucket once the historical data of the source bucket is migrated (replicated) to the target bucket, you can disable cross-region replication for the source bucket in the console.

  • Lightning Cube
    Not applicable.

  • OSS APIs/SDKs Copy Object and Upload Part Copy

    • Use of OssImport or cross-region replication is preferred, unless you have more refined or personalized requirements and a certain degree of capability in development.
    • Other relatively applicable situations:
      • The source and target buckets are in the same region.
      • Either the source or target bucket is of the Archive storage type.

Note: If the source and target buckets do not belong to the same user, OssImport takes priority over OSS APIs/SDKs, and cross-region replication is not applicable.

Others

If the problem persists, contact Alibaba Cloud After-Sales Technical Support.

Thank you! We've received your feedback.