When you need to migrate data from third-party cloud storage services, such as AWS S3 and Tencent Cloud COS, or from on-premises data centers to Alibaba Cloud OSS, traditional data migration methods often require complex development and have long transmission times. Data Online Migration provides a visual, unified migration platform that requires no development. You can use it to efficiently and securely migrate massive amounts of data from different sources to OSS with simple configurations.
How it works
Before you dive into this service, we recommend that you first understand what Object Storage Service (OSS) is.
Data Online Migration is a Data Transmission Service. Its core function is to create and execute a migration task. This task defines the data source, destination, and migration rules. Data Online Migration reads data from the source, transmits the data over the network, and writes the data to the specified OSS bucket.
Depending on the location of your source data and your network environment, Data Online Migration provides two migration modes: NAT mode and proxy mode.
Mode | Scenarios | How it works |
NAT mode | The source data is located in a third-party cloud storage service that can be accessed over the Internet, such as AWS S3, Tencent Cloud COS, or Huawei Cloud OBS. | Data Online Migration directly accesses and pulls data from the data address over the Internet and then writes the data to the destination OSS. |
Proxy mode | The source data is located in an environment that cannot be directly accessed over the Internet, such as an on-premises IDC, a local file system, or a scenario where a leased line or VPN is required for cloud migration. | Data Online Migration pulls data through an agent program deployed on the user side and then transmits the data to the destination OSS. |
Before you configure a migration task, you need to understand the following core concepts:
Data address: Specifies the access information for a data source, which can be the source or destination of a migration. A data address includes key information such as the data source type (for example, AWS S3), bucket, endpoint, and AccessKey pair. Data Online Migration uses the data address to access the corresponding data source.
Migration task: The core operational unit of Data Online Migration. By executing a migration task, you can migrate data from the source to the destination storage. When creating a task, you must specify the source and destination data addresses. You can also configure parameters such as traffic shaping, overwrite policies for files with the same name, auditing, and filters.
Proxy: A data bridge that connects your data source to Data Online Migration. It is an application that you deploy in your environment. A data address can be associated with multiple proxies within a channel. If a data address is associated with a proxy, Data Online Migration accesses the data source through that proxy.
Channel: In proxy mode, a channel is a logical concept that represents the data link between a proxy and Data Online Migration. A channel can contain multiple proxies, and each proxy belongs to a channel.
Important features
Supported data sources
Object Storage Service: Alibaba Cloud OSS, Amazon AWS S3, Tencent Cloud COS, Huawei Cloud OBS, Google Cloud GCS, Microsoft Azure BLOB, and more.
Local file system: Local disks, NAS/CPFS mounted on servers, and more.
HTTP manifest file: A checklist file that contains multiple URLs. Data Online Migration downloads these files in batches to migrate them.
Migration methods
NAT mode and proxy mode support different network scenarios such as the Internet and leased lines. You do not need to specify NAT mode or proxy mode. If your data address is associated with a proxy, proxy mode is used. Otherwise, NAT mode is used.
NAT mode
In NAT mode, Data Online Migration pulls data directly from the source storage over the Internet and writes it to the destination Alibaba Cloud OSS. You do not need to deploy any components in your environment during this procedure.
The following figure shows how to use NAT mode to migrate data from a third-party data source to Alibaba Cloud OSS:
Proxy mode
You must deploy a proxy in the same network environment as your source data. The proxy serves as a tunnel. Data Online Migration reads the source data through the proxy and writes it to Alibaba Cloud OSS.
The following figure shows how to use proxy mode to migrate data from a third-party data source to Alibaba Cloud OSS:
Migration configuration options
Configuration item | Description |
Traffic shaping | Limits the maximum bandwidth (Mbit/s) and the number of files migrated per second (QPS) for a task to prevent the migration from affecting your business. |
Handling of files with the same name | You can configure this to Do not overwrite, Overwrite all, or Overwrite based on last modified time.
|
File filter | Filters files to be migrated based on conditions such as file name and last modified time. |
Automatic scheduling | Supports scheduled or periodic execution to implement incremental migration or regular backups. |
Migration reports
After each task is executed, a detailed report is generated. The report includes a migration overview, such as the number and total size of migrated files, and the migration result for each file. This information facilitates auditing and verification.
Usage
You can manage Data Online Migration through the console. Log on to the Data Online Migration console. The console provides an intuitive and convenient GUI for all operations.
For batch and automated task management, such as creating multiple data addresses or migration tasks, you can use the SDKs provided by Alibaba Cloud. This lets you create data addresses and migration tasks and integrate Data Online Migration into your own management platform.
Service Agreement
For more information, see Data Transport Service Agreement.
The Data Transport Service Agreement includes terms related to Data Online Migration.
Related Services
In addition to Data Online Migration, Alibaba Cloud also provides an offline migration service. The offline migration service lets you use customized migration devices (Data Transport) to migrate terabytes or even petabytes of on-premises data to the cloud. This service is designed to address challenges related to the efficiency and security of large-scale data transmission. For more information, see Offline migration (Data Transport).