This topic describes the parameters that you can specify when you use Data Online Migration to migrate data.
Alibaba Cloud Object Storage Service
- OSS EndpointThe endpoint. The following table lists the valid formats of OSS endpoints.
No. Format Description 1 http://oss-cn-hangzhou.aliyuncs.com Upload or download data over the Internet by using HTTP. 2 http://oss-cn-hangzhou-internal.aliyuncs.com Upload or download data over the internal network by using HTTP. 3 https://oss-cn-hangzhou.aliyuncs.com Upload or download data over the Internet by using HTTPS. 4 https://oss-cn-hangzhou-internal.aliyuncs.com Upload or download data over the internal network by using HTTPS. For more information, see Regions and endpoints.
- OSS Bucket
The name of the bucket. The prefix or the suffix of the bucket name cannot contain invalid characters such as spaces, line feeds, and tab keys.
- OSS Prefix
The prefix. The files in the directory that have the specified prefix are migrated. If the name of a file is the same as the specified prefix, the file is not migrated. The prefix cannot start with a forward slash (/). However, the prefix must end with a forward slash (/). For example, you can specify the prefix as
docs/
. - AccessKey Id and AccessKey secret
The AccessKey pair that is used to migrate data. An AccessKey pair consists of an AccessKey ID and an AccessKey secret. You can use the AccessKey pair of an Alibaba Cloud account or a RAM user. You cannot use the AccessKey pair of a temporary user. For more information, see Access control.
If you migrate data from OSS, you can attach the following policy to a RAM user to grant the required permissions to the RAM user.If you migrate data to OSS, you can attach the following policy to a RAM user to grant the required permissions to the RAM user.You can create a RAM user for a data migration job, and use the AccessKey pair that is generated for the RAM user. After the migration job is completed, you can delete the AccessKey pair.
- Data Size and File Count
Specify the amount of data and the number of files that you want to migrate. You can view the storage usage and the number of the files in the OSS console.
AWS S3
- EndpointData Online Migration supports only the following AWS S3 regions and endpoints. These endpoints use HTTPS and HTTP protocols.
RegionName Region Endpoint US East (Ohio) us-east-2 http://s3.us-east-2.amazonaws.com http://s3-us-east-2.amazonaws.com
US East (N. Virginia) us-east-1 http://s3.us-east-1.amazonaws.com http://s3-us-east-1.amazonaws.com
US West (N. California) us-west-1 http://s3.us-west-1.amazonaws.com http://s3-us-west-1.amazonaws.com
US West (Oregon) us-west-2 http://s3.us-west-2.amazonaws.com http://s3-us-west-2.amazonaws.com
Canada (Central) ca-central-1 http://s3.ca-central-1.amazonaws.com http://s3-ca-central-1.amazonaws.com
Asia Pacific (Seoul) ap-northeast-2 http://s3.ca-central-1.amazonaws.com http://s3-ca-central-1.amazonaws.com
Asia Pacific (Osaka-Local) ap-northeast-3 http://s3.ap-northeast-2.amazonaws.com http://s3-ap-northeast-2.amazonaws.com
Asia Pacific (Singapore) ap-southeast-1 http://s3.ap-southeast-1.amazonaws.com http://s3-ap-southeast-1.amazonaws.com
Asia Pacific (Sydney) ap-southeast-2 http://s3.ap-southeast-2.amazonaws.com http://s3-ap-southeast-2.amazonaws.com
Asia Pacific (Tokyo) ap-northeast-1 http://s3.ap-northeast-1.amazonaws.com http://s3-ap-northeast-1.amazonaws.com
China (Beijing) cn-north-1 http://s3.cn-north-1.amazonaws.com.cn China (Ningxia) cn-northwest-1 http://s3.cn-northwest-1.amazonaws.com.cn Europe (Frankfurt) eu-central-1 http://s3.eu-central-1.amazonaws.com http://s3-eu-central-1.amazonaws.com
Europe (Ireland) eu-west-1 http://s3.eu-west-1.amazonaws.com http://s3-eu-west-1.amazonaws.com
Europe (London) eu-west-2 http://s3.eu-west-2.amazonaws.com http://s3-eu-west-2.amazonaws.com
Europe (Paris) eu-west-3 http://s3.eu-west-3.amazonaws.com http://s3-eu-west-3.amazonaws.com
South America (São Paulo) sa-east-1 http://s3.sa-east-1.amazonaws.com http://s3-sa-east-1.amazonaws.com
- Bucket
The name of the bucket. The prefix or the suffix of the bucket name cannot contain invalid characters such as spaces, line feeds, and tab keys.
- OSS Prefix
The prefix. The files in the directory that have the specified prefix are migrated. If the name of a file is the same as the specified prefix, the file is not migrated. The prefix cannot start with a forward slash (/). However, the prefix must end with a forward slash (/). For example, you can specify the prefix as
docs/
. - AccessKeyId and SecretAccessKey
The access key that is used to migrate data. The access key in AWS Identity and Access Management (IAM) consists of an access key ID and an access key secret. To obtain an access key, perform the following operations. Log on to the AWS IAM console, create an IAM user, attach the
AmazonS3ReadOnlyAccess
permission policy to the IAM user, and then create an access key for the IAM user. After the migration job is completed, you can delete the IAM user. - Data Size and File Count
Specify the amount of data and the number of files that you want to migrate. To ensure a successful migration, you must specify an appropriate data amount and the number of files based on an estimation of the data amount.
Azure Blob
- Storage Account, Secret, and Connection Strings
To view the details about the storage account, access keys, and connection strings, perform the following operations. Log on to the Microsoft Azure console. In the left-side navigation pane, click Storage accounts and select the storage account. In the Settings section, click Access keys. Then, you can view the details about the storage account, access keys, and connection strings.
- Container
The name of an Azure container.
- Prefix
The prefix. The files in the directory that have the specified prefix are migrated. If the name of a file is the same as the specified prefix, the file is not migrated. The prefix cannot start with a forward slash (/). However, the prefix must end with a forward slash (/). For example, you can specify the prefix as
docs/
. - Data Size and File Count
Specify the amount of data and the number of files that you want to migrate. An example is used to describe how to specify the two parameters. Log on to the Microsoft Azure console. In the left-side navigation pane, choose . On the container-property page, click Calculate Size to view the container size. Specify the two parameters based on the container size.
Tencent Cloud Object Storage
- RegionThe abbreviation of the region or the zone where the bucket is deployed. Tencent COS V4 supports the regions and zones that are listed in the following table. For more information, see Regions.
Region or zone Abbreviation Beijing Zone 1 tj Beijing bj Shanghai sh Guangzhou gz Chengdu cd Singapore sgp Hong Kong (China) hk Toronto ca Frankfurt ger - Bucket
The name of the COS bucket. The COS bucket name uses the
user-specified bucket name-APPID
format. APPID is a Tencent Cloud account identifier. An example of the COS bucket name istony-1234567890
. Note that the bucket name that you specify in the Data Transport console istony
. - Prefix
The prefix. The files in the directory that have the specified prefix are migrated. The prefix must start and end with a forward slash (/), for example,
/docs/
. - APPID
The APPID. To view the APPID, log on to the COS console and go to the Account Info page.
- Secret Id and Secret Key
The API key that is used to migrate data. An API key in COS consists of a SecretId and a SecretKey. To view or create an API key, log on to the COS console, choose . On the API Key Management page, you can view or create an API key. We recommend that you create an API key for a data migration job. We recommend that you delete the API key after the migration job is completed.
- Data Size and File Count
Specify the amount of data and the number of files that you want to migrate. Before you specify the two parameters, log on to the COS console, and choose . On the Object Storage page, choose . On the page that appears, you can view the amount of data and the number of objects that you want to migrate to the destination bucket.
Qiniu Cloud-Object Storage
- Endpoint
The endpoint. To view the endpoints, log on to the Qiniu Cloud console, navigate to the Object Storage page, and then select the bucket.
The endpoints use thehttp://<Domain name>
format.http://oy4jki81y.bkt.clouddn.com http://78rets.com1.z0.glb.clouddn.com http://cartoon.u.qiniudn.com
Note Only the endpoints of the three types in this figure are valid. - Bucket
The name of the KODO bucket.
- Prefix
The prefix. The files in the directory that have the specified prefix are migrated. If the name of a file is the same as the specified prefix, the file is not migrated. The prefix cannot start with a forward slash (/). However, the prefix must end with a forward slash (/). For example, you can specify the prefix as
docs/
. - Access Key and Secret KeyThe Access Key and the Secret Key that are used to migrate data. To view the Access Key and the Secret Key, log on to the Qiniu Cloud console and choose .Note You must specify an Access Key and a Secret Key that are in use.
- Data Size and File Count
Specify the amount of data and the number of files that you want to migrate. Before you specify the two parameters, perform the following operations. Log on to the Qiniu Cloud console, navigate to the Object Storage page, and then select the bucket. On the page that appears, click the Content Management tab.
Baidu Object Storage
- EndpointThe endpoint. You can use Data Transport to migrate data from the BOS service that is deployed in the China (Beijing), China (Guangzhou), or China (Suzhou) region. The following table lists the BOS endpoints that are supported by Data Online Migration.
Abbreviation Domain name Protocol Endpoint North China - Beijing bj.bcebos.com HTTP or HTTPS http://bj.bcebos.com South China - Guangzhou gz.bcebos.com http://gz.bcebos.com East China - Suzhou su.bcebos.com http://su.bcebos.com To view the region where the bucket is deployed, log on to the BOS console, and choose .
- Prefix
The prefix. The files in the directory that have the specified prefix are migrated. If the name of a file is the same as the specified prefix, the file is not migrated. The prefix cannot start with a forward slash (/). However, the prefix must end with a forward slash (/). For example, you can specify the prefix as
docs/
. - Access Key ID and Secret Access Key
The Access Key ID and the Secret Access Key that are used to migrate data. To view the Access Key ID and the Secret Access Key, log on to the BOS console and choose .
- Data Size and File Count
Specify the amount of data and the number of files that you want to migrate. Before you specify the two parameters, perform the following operations. Log on to the BOS console and choose . On the page that appears, view the storage usage of the bucket. Estimate the number of files that you want to migrate based on the storage usage. To ensure a successful migration, you must specify an appropriate data amount and the number of files based on an estimation of the data amount.
HTTP HTTPS
- File PathThe file path. Note that a list file has two columns.
- The first column includes a list of HTTP and HTTPS URLs. You must encode the special characters that are included in the URLs. Data Transport uses the HTTP GET method to download a file and use the HTTP HEAD method to obtain the file metadata from an HTTP or HTTPS URL.
- The second column includes a list of file names. After a file is migrated, the name
of the corresponding object consists of a prefix and the file name. Separate the two
columns with a tab key (
\t
).
\n
). The following examples show the valid format.http://127.0.0.1/docs/my.doc docs/my.doc http://127.0.0.1/pics/my.jpg pics/my.jpg http://127.0.0.1/exes/my.exe exes/my.exe
File path format
After the migration file is generated, upload it to Alibaba Cloud OSS. Data Transport downloads the list file and migrates the files based on the specified URLs. The file path uses the oss://{Bucket}/{The name of the list file} format, as shown in the following example.oss://mybucket/httplist.txt
- List Access EndpointThe endpoint. The following table lists the valid formats of OSS endpoints.
No. Format Description 1 http://oss-cn-hangzhou.aliyuncs.com Upload or download data over the Internet by using HTTP. 2 http://oss-cn-hangzhou-internal.aliyuncs.com Upload or download data over the internal network by using HTTP. 3 https://oss-cn-hangzhou.aliyuncs.com Upload or download data over the Internet by using HTTP. 4 https://oss-cn-hangzhou-internal.aliyuncs.com Upload or download data over the internal network by using HTTP. - List Access AK and List Access SK
The credentials that are used to download the list file. You can use the credentials of the primary account or a sub-user. If you use the credentials of a sub-user, you must grant the GetObject permissions to the sub-user.
- Data Size and File Count
Specify the amount of data and the number of files that you want to migrate. To ensure a successful migration, you must specify an appropriate data amount and the number of files based on an estimation of the data amount.
USS
- Domain AddressThe USS endpoint that is used as an access point for SDKs, or the domain name in a request URL when you call a RESTful API operation. You can use one of the endpoints that are listed in the following table.Note To view the network type, visit http://v0.api.upyun.com.
Network Type Endpoint Description Intelligent Routing http://v0.api.upyun.com Recommended China Telecom http://v1.api.upyun.com N/A China Unicom or China Netcom http://v2.api.upyun.com N/A China Mobile (Tietong) http://v3.api.upyun.com N/A - Service Name
The service name. To view the service name, log on to the UPYUN console, and navigate to the UPYUN Storage Service page.
- Migration Folder
The folder in which the files are migrated. The folder is specified by a prefix. The prefix cannot start with a forward slash (/). However, the prefix must end with a forward slash (/). For example, you can specify the prefix as docs/.
- Operator Name and Operator Secret
The name of the operator. To view the operator details or add an operator, log on to the UPYUN Cloud console, click your account name in the upper-right corner of the page, and select Account Management from the drop-down list. On the page that appears, you can view the operator details or add an operator. You can add an operator for a data migration job and specify a corresponding password for the operator. To allow the operator to migrate data, you must grant the operator the read permissions.
To grant the operator the required permissions, perform the following operations. On the UPYUN Storage Service page, click the name of the bucket, navigate through , and then click Authorize. In the dialog box that appears, select the required operator and click OK.
- Data Size and File Count
Specify the amount of data and the number of files that you want to migrate. Before you specify the two parameters, log on to the UPYUN console and navigate through . On the page that appears, view the amount of data and the number of files stored in the bucket. Estimate the amount of data and the number of files that you want to migrate based on the results. To ensure a successful migration, you must specify an appropriate data amount and the number of files based on an estimation of the data amount.
Kingsoft Standard Storage Service
- EndpointThe following table lists the mappings between KS3 regions and endpoints.
Region Endpoint BEIJING ks3-cn-beijing.ksyun.com SHANGHAI ks3-cn-shanghai.ksyun.com HONGKONG ks3-cn-hk-1.ksyun.com China (Hangzhou) kss.ksyun.com RUSSIA ks3-rus.ksyun.com SINGAPORE ks3-sgp.ksyun.com Note To view the region where the bucket is deployed, log on to the KS3 console, and navigate through . - Bucket
The name of the KS3 bucket.
- Prefix
The prefix. The files in the directory that have the specified prefix are migrated. If the name of a file is the same as the specified prefix, the file is not migrated. The prefix cannot start with a forward slash (/). However, the prefix must end with a forward slash (/). For example, you can specify the prefix as
docs/
. - Access Key ID and Secret Key
The Access Key ID and the Secret Access Key that are used to migrate data. To create an Access Key ID and a Secret Access Key, log on to the KS3 console, navigate through . On the page that appears, create an IAM user, and attach the
KS3FullAccess
policy to the IAM user to grant the IAM user the required permissions. On the IAM user details page, create an AccessKey ID and a Secret Access Key. You can delete the IAM user after the migration job is completed. - Data Size and File Count
Specify the amount of data and the number of files that you want to migrate. Before you specify the two parameters, log on to the KS3 console and navigate through . On the page that appears, view the amount of data and the number of files stored in the bucket. Estimate the amount of data and the number of files that you want to migrate based on the results. To ensure a successful migration, you must specify an appropriate data amount and the number of files based on an estimation of the data amount.
Huawei Object Storage Service
- EndpointThe following table lists the regions that are supported by Huawei OBS. For more information, see Regions and endpoints.
Region name Abbreviation Endpoint Protocol type CN North-Beijing4 cn-north-4 obs.cn-north-4.myhuaweicloud.com HTTPS and HTTP CN North-Beijing1 cn-north-1 obs.cn-north-1.myhuaweicloud.com CN East-Shanghai2 cn-east-2 obs.cn-east-2.myhuaweicloud.com CN East-Shanghai1 cn-east-3 obs.cn-east-3.myhuaweicloud.com CN South-Guangzhou cn-south-1 obs.cn-south-1.myhuaweicloud.com CN Southwest-Guiyang1 cn-southwest-2 obs.cn-southwest-2.myhuaweicloud.com AP-Bangkok ap-southeast-2 obs.ap-southeast-2.myhuaweicloud.com AP-Hong Kong ap-southeast-1 obs.ap-southeast-1.myhuaweicloud.com AP-Singapore ap-southeast-3 obs.ap-southeast-3.myhuaweicloud.com AF-Johannesburg af-south-1 obs.af-south-1.myhuaweicloud.com Note To view the region where the bucket is deployed, log on to the KS3 console, and navigate through . - Bucket
The name of the Huawei OBS bucket.
- Prefix
The prefix. The files in the directory that have the specified prefix are migrated. If the name of a file is the same as the specified prefix, the file is not migrated. The prefix cannot start with a forward slash (/) and must end with a forward slash (/). For example, you can specify the prefix as
docs/
. - Access Key ID and Secret Access Key
The access key that is used to migrate data. An access key in Huawei Cloud consists of an access key ID and an access key secret. To view or create an access key, log on to the Huawei Cloud console. In the upper-right corner of the console, move the pointer over the account name and click My Credentials. On the page that appears, navigate through . If you have not specified an access key during the first logon, click Create Access Key to create an access key.
- Data Size and File Count
Specify the amount of data and the number of files that you want to migrate. Before you specify the two parameters, perform the following operations. Log on to the Huawei Cloud console, choose , and then select the bucket that you want to migrate. In the Basic Statistics section, check the size of the bucket and the number of files that you want to migrate.
UCloud UFile
- RegionThe region where the UCloud UFile service is deployed. UCloud UFile supports the regions that are listed in the following table. For more information, see the Regions and zones.
Region name Abbreviation China (Beijing) cn-bj China (Hong Kong) hk China (Guangzhou) cn-gd China (Shanghai 2) cn-sh2 US (Los Angeles) us-ca Singapore (Singapore) sg Indonesia (Jakarta) idn-jakarta Nigeria (Lagos) afr-nigeria Brazil (Sao Paulo) bra-saopaulo UAE (Dubai) uae-dubai Vietnam (Ho Chi Minh City) vn-sng China (Taipei) tw-tp India (Mumbai) ind-mumbai US (Washington) us-ws Germany (Frankfurt) ge-fra - Bucket
The name of the UFile bucket.
- Prefix
The prefix. The files in the directory that have the specified prefix are migrated. If the name of a file is the same as the specified prefix, the file is not migrated. The prefix cannot start with a forward slash (/). However, the prefix must end with a forward slash (/). For example, you can specify the prefix as
docs/
. - Public Key and Private Key
The API public key and the API private key that are used to migrate data. To view the API public key and the API private key, log on to the UCloud console. Then, navigate to the API key management page, and view the API public key and the API private key.
- Data Size and File Count
Specify the amount of data and the number of files that you want to migrate. Before you specify the two parameters, perform the following operations. Log on to the UCloud console and navigate to the UFile page. On this page, click Bucket and click the name of the bucket that stores the source data. On the Overview tab, view the storage usage of the target bucket. Estimate the amount of data and the number of files that you want to migrate on the storage usage. To ensure a successful migration, you must specify an appropriate data amount and the number of files based on an estimation of the data amount.
GCP
This section describes the parameters for Google Cloud Platform (GCP). You may need to specify these parameters when you use the Data Transport console to create data migration jobs.
- Bucket
The name of the bucket.
- Prefix
The prefix. The files in the directory that have the specified prefix are migrated. If the name of a file is the same as the specified prefix, the file is not migrated. The prefix cannot start with a forward slash (/). However, the prefix must end with a forward slash (/). For example, you can specify the prefix as docs/.
- Key File
- Log on to the GCP console.
- In the left-side navigation pane, click Service Accounts.
- On the Service Accounts page, find the service account. In the Actions column, choose .
- In the Create private key for "username" dialog box, select JSON as the key type, and click CREATE.
- GCP automatically downloads a JSON key file. Upload the key file to the Data Transport console.
- Data Size and File Count
Specify the amount of data and the number of files that you want to migrate. To ensure a successful migration, you must specify an appropriate data amount and the number of files based on an estimation of the data amount.