Parameter description for data migration jobs - Parameter description| Alibaba Cloud Documentation Center

Alibaba Cloud Object Storage Service

This section describes the parameters for Alibaba Cloud Object Storage Service (OSS). You may need to specify these parameters when you use the Data Transport console to create data migration jobs.

OSS Endpoint

The OSS endpoint. The following table lists the valid formats of the OSS endpoints.


No.	Format	Description
1	http://oss-cn-hangzhou.aliyuncs.com	Upload or download data by using HTTP over the Internet.
2	http://oss-cn-hangzhou-internal.aliyuncs.com	Upload or download data by using HTTP over the internal network.
3	https://oss-cn-hangzhou.aliyuncs.com	Upload or download data by using HTTP over the Internet.
4	https://oss-cn-hangzhou-internal.aliyuncs.com	Upload or download data by using HTTP over the internal network.

If you migrate data from OSS, Data Transport automatically downloads data over the internal network. If you migrate data to OSS, OSS uploads data over the internal network. The following formats are invalid. OSS_Endpoint

For more information about OSS endpoints, see Regions and endpoints.

OSS Bucket
The name of the bucket. The prefix or the suffix of the bucket name cannot contain invalid characters such as spaces, line feeds, and tab keys.
OSS Prefix
The OSS prefix. The files in the directory that has the specified prefix are to be migrated. If the name of a file is the same as the specified prefix, the file is not to be migrated. The prefix cannot start with a forward slash (/) and must end with a forward slash (/). For example, you can specify the prefix as docs/.
AccessKey Id and AccessKey Secret
The AccessKey pair that is used to migrate data. An AccessKey pair in Alibaba Cloud Resource Management Access (RAM) consists of an AccessKey ID and an AccessKey secret. You can use the AccessKey pair of an Alibaba Cloud account or a RAM user. You cannot use the AccessKey pair of a temporary user. For more information, see Access control.

If you migrate data from OSS, you can attach the following policy to a RAM user to grant the RAM user the required permissions.

If you migrate data to OSS, you can attach the following policy to a RAM user to grant the RAM user the required permissions.

You can create a RAM user for a data migration job, and use the AccessKey pair that is generated for the RAM user. After the data migration is complete, you can delete the AccessKey pair.
Data Size and File Count
Specify the amount of data and the number of files to be migrated in the OSS bucket. You can view the storage usage and the number of the files for the target OSS bucket in the OSS console.

AWS S3

This section describes the parameters for AWS Simple Storage Service (AWS S3). You may need to specify these parameters when you use the Data Transport console to create data migration jobs.

Note Currently, AWS S3 whitelists are not supported by Data Transport. If whitelists are configured in AWS S3, delete the whitelists or submit a ticket before data migration.

Endpoint

Data Transport supports only the following AWS S3 regions and endpoints. These endpoints use HTTPS and HTTP protocols.


Region name	Region	Endpoint
US East (Ohio)	us-east-2	http://s3.us-east-2.amazonaws.com http://s3-us-east-2.amazonaws.com
US East (N. Virginia)	us-east-1	http://s3.us-east-1.amazonaws.com http://s3-us-east-1.amazonaws.com
US West (N. California)	us-west-1	http://s3.us-west-1.amazonaws.com http://s3-us-west-1.amazonaws.com
US West (Oregon)	us-west-2	http://s3.us-west-2.amazonaws.com http://s3-us-west-2.amazonaws.com
Canada (Central)	ca-central-1	http://s3.ca-central-1.amazonaws.com http://s3-ca-central-1.amazonaws.com
Asia Pacific (Seoul)	ap-northeast-2	http://s3.ca-central-1.amazonaws.com http://s3-ca-central-1.amazonaws.com
Asia Pacific (Osaka-Local)	ap-northeast-3	http://s3.ap-northeast-2.amazonaws.com http://s3-ap-northeast-2.amazonaws.com
Asia Pacific (Singapore)	ap-southeast-1	http://s3.ap-southeast-1.amazonaws.com http://s3-ap-southeast-1.amazonaws.com
Asia Pacific (Sydney)	ap-southeast-2	http://s3.ap-southeast-2.amazonaws.com http://s3-ap-southeast-2.amazonaws.com
Asia Pacific (Tokyo)	ap-northeast-1	http://s3.ap-northeast-1.amazonaws.com http://s3-ap-northeast-1.amazonaws.com
China (Beijing)	cn-north-1	http://s3.cn-north-1.amazonaws.com.cn
China (Ningxia)	cn-northwest-1	http://s3.cn-northwest-1.amazonaws.com.cn
Europe (Frankfurt)	eu-central-1	http://s3.eu-central-1.amazonaws.com http://s3-eu-central-1.amazonaws.com
Europe (Ireland)	eu-west-1	http://s3.eu-west-1.amazonaws.com http://s3-eu-west-1.amazonaws.com
Europe (London)	eu-west-2	http://s3.eu-west-2.amazonaws.com http://s3-eu-west-2.amazonaws.com
Europe (Paris)	eu-west-3	http://s3.eu-west-3.amazonaws.com http://s3-eu-west-3.amazonaws.com
South America (São Paulo)	sa-east-1	http://s3.sa-east-1.amazonaws.com http://s3-sa-east-1.amazonaws.com

Bucket
The name of the bucket. The prefix or the suffix of the bucket name cannot contain invalid characters such as spaces, line feeds, and tab keys.
OSS Prefix
The OSS prefix. The files in the directory that has the specified prefix are to be migrated. If the name of a file is the same as the specified prefix, the file is not to be migrated. The prefix cannot start with a forward slash (/) and must end with a forward slash (/). For example, you can specify the prefix as docs/.
AccessKeyId and SecretAccessKey
The access key that is used to migrate data. The access key in AWS Identity and Access Management (IAM) consists of an access key ID and a secret access key. To obtain an access key, perform the following operations. Log on to the AWS IAM console, create an IAM user, attach the AmazonS3ReadOnlyAccess permission policy to the IAM user, and then create an access key for the IAM user. After the data migration is complete, you can delete the IAM user.
Data Size and File Count
Specify the amount of data and the number of files to be migrated. To ensure a successful migration, you must specify an appropriate data size and the number of files based on an accurate estimation of the actual data amount.

Azure Blob

This section describes the parameters for Microsoft Azure Blob. You may need to specify these parameters when you use the Data Transport console to create data migration jobs.

Storage Account, Secret, and Connection Strings
To view the details about the storage account, access keys, and connection strings, perform the following operations. Log on to the Microsoft Azure console. In the left-side navigation pane, click Storage accounts, and select the target storage account. In the Settings section, click Access keys. Then, you can view the details about the storage account, access keys, and connection strings.
Container
The name of an Azure container.
Prefix
The prefix. The files in the directory that has the specified prefix are to be migrated. If the name of a file is the same as the specified prefix, the file is not to be migrated. The prefix cannot start with a forward slash (/) and must end with a forward slash (/). For example, you can specify the prefix as docs/.
Data Size and File Count
Specify the amount of data and the number of files to be migrated. An example is used to describe how to specify the two parameters. Log on to the Microsoft Azure console. In the left-side navigation pane, choose testbucket2018 - Blob > container - Property. On the container-property page, click Calculate Size to view the container size. Based on the container size, specify the two parameters.

Tencent Cloud Object Storage

Data Transport supports Tencent Cloud Object Storage (COS) V4 and uses the COS API of version 4 to access the COS service. This section describes the parameters for Tencent COS. You may need to specify these parameters when you use the Data Transport console to create data migration jobs.

Region

The abbreviation of the region or the zone where the target bucket is deployed. Tencent COS V4 supports the regions and zones that are listed in the following table. For more information, see Regions.


Region or zone	Abbreviation
Beijing Zone 1	tj
North China (Beijing)	bj
East China (Shanghai)	sh
South China (Guangzhou)	gz
Southwest China (Chengdu)	cd
Southeast Asia Pacific (Singapore)	sgp
Hong Kong/Macao/Taiwan (Hong Kong, China)	hk
North America (Toronto)	ca
Europe (Frankfurt)	ger

Bucket
The name of the COS bucket. The COS bucket name uses the user-specified bucket name-APPID format. APPID is a Tencent Cloud account identifier. An example of the COS bucket name is tony-1234567890. Note that the bucket name that you specify in the Data Transport console is tony.
Prefix
The prefix. The files in the directory that has the specified prefix are to be migrated. The prefix must start and end with a forward slash (/), for example, /docs/.
APPID
The APPID. To view the APPID, log on to the COS console and navigate to the Account Info page.
Secret Id and Secret Key
The API key that is used to migrate data. An API key in COS consists of a SecretId and a SecretKey. To view or create an API key, log on to the COS console, choose Cloud Services > Management Tools > API Key Management. On the API Key Management page, you can view or create an API key. We recommend that you create an API key for a data migration job. We recommend that you delete the API key after the data migration is complete.
Data Size and File Count
Specify the amount of data and the number of files to be migrated. Before you specify the two parameters, perform the following operations. Log on to the COS console, and choose Cloud Services > Storage > Cloud Object Storage. On the Object Storage page, choose Monitoring Report > Basic Statistics. On the page that appears, you can view the amount of data and the number of objects to be migrated in the target bucket.

Qiniu Cloud-Object Storage

This section describes the parameters for Qiniu Cloud-Object Storage (KODO). You may need to specify these parameters when you use the Data Transport console to create data migration jobs.

Endpoint
The endpoint. To view the endpoints, log on to the Qiniu Cloud console, navigate to the Object Storage page, and then select the target bucket. On the page that appears, you can view the endpoints in the Converged CDN Test Domain and the Converged CDN Accelerated Domain sections.
The endpoints use the http://<Domain name> format, as shown in the following figure.
```
http://oy4jki81y.bkt.clouddn.com
http://78rets.com1.z0.glb.clouddn.com
http://cartoon.u.qiniudn.com
```
Note Only the endpoints of the three types in this figure are valid.
Bucket
The name of the KODO bucket.
Prefix
The prefix. The files in the directory that has the specified prefix are to be migrated. If the name of a file is the same as the specified prefix, the file is not to be migrated. The prefix cannot start with a forward slash (/) and must end with a forward slash (/). For example, you can specify the prefix as docs/.
Access Key and Secret Key
The Access Key and the Secret Key that are used to migrate data. To view the Access Key and the Secret Key, log on to the Qiniu Cloud console and choose Personal Panel > Key Management.

Note You must specify an Access Key and a Secret Key that are in use.
Data Size and File Count
Specify the amount of data and the number of files to be migrated. Before you specify the two parameters, perform the following operations. Log on to the Qiniu Cloud console, navigate to the Object Storage page, and then select the target bucket. On the page that appears, click the Content Management tab. On this tab, you can view the number of files and the amount of data stored in the bucket.

Baidu Object Storage

This section describes the parameters for Baidu Object Storage (BOS). You may need to specify these parameters when you use the Data Transport console to create data migration jobs.

Endpoint

The endpoint. Currently, you can use Data Transport to migrate data from the BOS service that is deployed in the China (Beijing), the China (Guangzhou), or the China (Suzhou) region. The following table lists the BOS endpoints that are supported by Data Transport.


Region	Domain name	Protocol	Endpoint
China (Beijing)	bj.bcebos.com	HTTP or HTTPS	http://bj.bcebos.com
China (Guangzhou)	gz.bcebos.com		http://gz.bcebos.com
China (Suzhou)	su.bcebos.com		http://su.bcebos.com

To view the region where the bucket is deployed, log on to the BOS console, and choose Cloud Services > BOS-Bucket Management.

Prefix
The prefix. The files in the directory that has the specified prefix are to be migrated. If the name of a file is the same as the specified prefix, the file is not to be migrated. The prefix cannot start with a forward slash (/) and must end with a forward slash (/). For example, you can specify the prefix as docs/.
Access Key ID and Secret Access Key
The Access Key ID and the Secret Access Key that are used to migrate data. To view the Access Key ID and the Secret Access Key, log on to the BOS console and choose Authorization > Access Key.
Data Size and File Count
Specify the amount of data and the number of files to be migrated. Before you specify the two parameters, perform the following operations. Log on to the BOS console and choose Cloud Services > BOS-Bucket Management > Bucket Overview. On the page that appears, view the storage usage of the target bucket. Based on the storage usage, estimate the number of files to be migrated. To ensure a successful migration, you must specify the number of files based on an accurate estimation.

HTTP and HTTPS

This section describes the parameters for HTTP and HTTPS sources. You may need to specify these parameters when you use the Data Transport console to create data migration jobs.

File Path
The file path. Note that a list file has two columns.
- The first column includes a list of HTTP and HTTPS URLs. You must encode the special characters that are included in the URLs. Data Transport uses the HTTP GET method to download a file and use the HTTP HEAD method to obtain the file metadata from an HTTP or HTTPS URL, respectively.
- The second column includes a list of file names. After a file is migrated, the name of the corresponding object consists of a prefix and the file name. Separate the two columns with a tab key (\t).
Each line includes a file name. Separate lines with a line feed (\n). The following examples show the valid format.
```
http://127.0.0.1/docs/my.doc    docs/my.doc
http://127.0.0.1/pics/my.jpg    pics/my.jpg
http://127.0.0.1/exes/my.exe    exes/my.exe
```
File path format
After the migration file is generated, upload it to Alibaba Cloud OSS. Data Transport downloads the list file and migrates the files based on the specified URLs. The file path uses the oss://{Bucket}/{The name of the list file} format, as shown in the following example.
```
oss://mybucket/httplist.txt
```

List Access Endpoint

The endpoint. The following table lists the valid formats of the OSS endpoints.


No.	Format	Description
1	http://oss-cn-hangzhou.aliyuncs.com	Upload or download data by using HTTP over the Internet.
2	http://oss-cn-hangzhou-internal.aliyuncs.com	Upload or download data by using HTTP over the internal network.
3	https://oss-cn-hangzhou.aliyuncs.com	Upload or download data by using HTTP over the Internet.
4	https://oss-cn-hangzhou-internal.aliyuncs.com	Upload or download data by using HTTP over the internal network.

For more information about OSS endpoints, see Regions and endpoints.

List Access AK and List Access SK
The credentials that are used to download the list file. You can use the credentials of the primary account or a subuser. If you use the credentials of a subuser, you must grant the subuser the GetObject permissions.
Data Size and File Count
Specify the amount of data and the number of files to be migrated. To ensure a successful migration, you must specify an appropriate data size and the number of files based on an accurate estimation of the actual data amount.

UPYUN Storage Service

This topic describes the parameters for UPYUN Storage Service (USS). You may need to configure these parameters when you use the Data Transport service.

Domain Address

The USS endpoint that is used as an access point for SDKs, or the domain name in a request URL when you call a RESTful API operation. You can use one of the endpoints that are listed in the following table.

Note To view the network type, visit http://v0.api.upyun.com.


Network type	Endpoint	Description
Intelligent routing	http://v0.api.upyun.com	Recommended
China Telecom	http://v1.api.upyun.com	N/A
China Unicom or China Netcom	http://v2.api.upyun.com	N/A
China Mobile or China Mobile Tietong	http://v3.api.upyun.com	N/A

Service Name
The service name. To view the service name, log on to the UPYUN console, and navigate to the UPYUN Storage Service page.
Migration Folder
The folder in which the files are to be migrated. The folder is specified by a prefix. The prefix cannot start with a forward slash (/) and must end with a forward slash (/). For example, you can specify the prefix as docs/.
Operator Name and Operator Secret
The name of the operator. To view the operator details or add an operator, log on to the UPYUN Cloud console, click your account name in the upper-right corner, and select Account Management from the drop-down list. On the page that appears, you can view the operator details or add an operator. You can add an operator for a data migration job and specify a corresponding password for the operator. To allow the operator to migrate data, you must grant the operator the read permissions.

To grant the operator the required permissions, perform the following operations. On the UPYUN Storage Service page, click the name of the bucket to be migrated, navigate through Configurations > Content Management, and then click Authorize. In the dialog box that appears, select the required operator and click OK.
Data Size and File Count
Specify the amount of data and the number of files to be migrated. Before you specify the two parameters, perform the following operations. Log on to the UPYUN console and navigate through UPYUN Storage Service > Configurations. On the page that appears, view the amount of data and the number of files stored in the bucket. Based on the results, estimate the amount of data and the number of files to be migrated. To ensure a successful migration, you must specify the parameters based on an accurate estimation.

Kingsoft Standard Storage Service

This section describes the parameters for Kingsoft Standard Storage Service (KS3). You may need to configure these parameters when use Data Transport to migrate data.

Endpoint

The following table lists the mappings between KS3 regions and endpoints.


Region	Endpoint
China (Beijing)	ks3-cn-beijing.ksyun.com
China (Shanghai)	ks3-cn-shanghai.ksyun.com
China (Hong Kong)	ks3-cn-hk-1.ksyun.com
China (Hangzhou)	kss.ksyun.com
Russia	ks3-rus.ksyun.com
Singapore	ks3-sgp.ksyun.com

Note To view the region where the bucket is deployed, log on to the KS3 console, and navigate through Storage & CDN > Object Storage > My Buckets > Bucket Details.

Bucket
The name of the KS3 bucket.
Prefix
The prefix. The files in the directory that has the specified prefix are to be migrated. If the name of a file is the same as the specified prefix, the file is not to be migrated. The prefix cannot start with a forward slash (/) and must end with a forward slash (/). For example, you can specify the prefix as docs/.
Access Key ID and Secret Key
The Access Key ID and the Secret Access Key that are used to migrate data. To create an Access Key ID and a Secret Access Key, log on to the KS3 console, navigate through Identities and Management > User Management. On the page that appears, create an IAM user, and attach the KS3FullAccess policy to the IAM user to grant the IAM user the required permissions. On the IAM user details page, create an AccessKey ID and a Secret Access Key. You can delete the IAM user after the data migration is complete.
Data Size and File Count
Specify the amount of data and the number of files to be migrated. Before you specify the two parameters, perform the following operations. Log on to the KS3 console and navigate through Storage & CDN > Object Storage > My Buckets > Bucket Details. On the page that appears, view the amount of data and the number of files stored in the bucket. Based on the results, estimate the amount of data and the number of files to be migrated. To ensure a successful migration, you must specify the parameters based on an accurate estimation.

Huawei Object Storage Service

This section describes the parameters for Huawei Object Storage Service (OBS). You may need to specify these parameters when you use the Data Transport console to create data migration jobs.

Endpoint

The following table lists the regions that are supported by Huawei OBS. For more information, see Regions and endpoints.


Region name	Abbreviation	Endpoint	Protocol type
CN North-Beijing4	cn-north-4	obs.cn-north-4.myhuaweicloud.com	HTTPS and HTTP
CN North-Beijing1	cn-north-1	obs.cn-north-1.myhuaweicloud.com
CN East-Shanghai2	cn-east-2	obs.cn-east-2.myhuaweicloud.com
CN East-Shanghai1	cn-east-3	obs.cn-east-3.myhuaweicloud.com
China (Guangzhou)	cn-south-1	obs.cn-south-1.myhuaweicloud.com
CN South-Guangzhou	cn-southwest-2	obs.cn-southwest-2.myhuaweicloud.com
AP-Bangkok	ap-southeast-2	obs.ap-southeast-2.myhuaweicloud.com
AP-Hong Kong	ap-southeast-1	obs.ap-southeast-1.myhuaweicloud.com
AP-Singapore	ap-southeast-3	obs.ap-southeast-3.myhuaweicloud.com
AF-Johannesburg	af-south-1	obs.af-south-1.myhuaweicloud.com

Bucket
The name of the Huawei OBS bucket.
Prefix
The prefix. The files in the directory that has the specified prefix are to be migrated. If the name of a file is the same as the specified prefix, the file is not to be migrated. The prefix cannot start with a forward slash (/) and must end with a forward slash (/). For example, you can specify the prefix as docs/.
Access Key ID and Secret Access Key
The access key that is used to migrate data. An access key in Huawei Cloud consists of an access key ID and a secret access key. To view or create an access key, log on to the Huawei Cloud console. In the upper-right corner of the console, move the pointer over the account name and click My Credentials. On the page that appears, navigate through My Credentials > Access Keys > Create Access Key. If you have not specified an access key during the first logon, click Create Access Key to create an access key.
Data Size and File Count
Specify the amount of data and the number of files to be migrated. Before you specify the two parameters, perform the following operations. Log on to the Huawei Cloud console, choose Service List > Object Storage Service > Object Storage, and then select the bucket to be migrated. In the Basic Statistics section, view the storage usage and the number of the files (objects) to be migrated in the bucket. The following figure shows an example.

UCloud UFile

The section describes the parameters for UCloud UFile. You may need to specify these parameters when you use the Data Transport console to create data migration jobs.

Region

The region where the UCloud UFile service is deployed. UCloud UFile supports the regions that are listed in the following table. For more information, see the Regions and zones.


Region name	Abbreviation
China (Beijing)	cn-bj
China (Hong Kong)	hk
China (Guangzhou)	cn-gd
China (Shanghai 2)	cn-sh2
US (Los Angeles)	us-ca
Singapore	sg
Indonesia (Jakarta)	idn-jakarta
Nigeria (Lagos)	afr-nigeria
Brazil (Sao Paulo)	bra-saopaulo
Dubai	uae-dubai
Vietnam (Ho Chi Minh City)	vn-sng
China (Taipei)	tw-tp
India (Mumbai)	ind-mumbai
US (Washington)	us-ws
Germany (Frankfurt)	ge-fra

Bucket
The name of the UFile bucket.
Prefix
The prefix. The files in the directory that has the specified prefix are to be migrated. If the name of a file is the same as the specified prefix, the file is not to be migrated. The prefix cannot start with a forward slash (/) and must end with a forward slash (/). For example, you can specify the prefix as docs/.
Public Key and Private Key
The API public key and the API private key that are used to migrate data. To view the API public key and the API private key, log on to the UCloud console. Then, navigate to the API key management page, and view the API public key and the API private key.
Data Size and File Count
Specify the amount of data and the number of files to be migrated. Before you specify the two parameters, perform the following operations. Log on to the UCloud console and navigate to the UFile page. On this page, click Bucket and click the name of the target bucket that stores the source data. On the Overview tab, view the storage usage of the target bucket. Based on the storage usage, estimate the amount of data and the number of files to be migrated. To ensure a successful migration, you must specify the number of files based on an accurate estimation.