This topic describes how to run the cp command to download objects from Object Storage Service (OSS) to your on-premises device.
Command syntax
./ossutil64 cp cloud_url file_url
[-r, --recursive]
[-f --force]
[-u --update]
[--maxdownspeed <value>]
[--disable-ignore-error]
[--only-current-dir]
[--bigfile-threshold <value>]
[--part-size <value>]
[--checkpoint-dir <value>]
[--range <value>]
[--encoding-type <value>]
[--include <value>]
[--exclude <value>]
[--meta <value>]
[--acl <value>]
[--snapshot-path <value>]
[--disable-crc64]
[--payer <value>]
[--partition-download <value>]
[-j, --job <value>]
[--parallel <value>]
[--version-id <value>]The following table describes the parameters that you can configure to run the cp command.
| Parameter | Description |
|---|---|
| cloud_url | The path of the object that you want to download from OSS. Format: oss://bucketname/objectname. Example: oss://examplebucket/examplefile.txt. |
| file_url | The local path to which you want to download the object from OSS. Examples: /localfolder/examplefile.txt on Linux or D:\localfolder\examplefile.txt on Windows. |
| -r, --recursive | The recursive operations. If you configure this parameter, ossutil performs operations on all objects in a bucket that meet the specified condition. If you do not configure this parameter, ossutil performs operations only on the specified object. |
| -f --force | Specifies that the operation is forcibly performed. The operation is performed without a prompt for confirmation. |
| -u, --update | Specifies that ossutil downloads an object from OSS only when the destination file does not exist in the local device or when the last modification time of the source object is later than that of the destination file. |
| --maxdownspeed | The maximum download speed. Unit: KB/s. The default value is 0, which indicates that the download speed is unlimited. |
| --disable-ignore-error | Specifies that errors are not ignored during batch operations. |
| --only-current-dir | Specifies that only objects in the current directory are downloaded. Subdirectories in the current directory and objects in these subdirectories are not downloaded. |
| --bigfile-threshold | The object size threshold for using resumable download. If an object exceeds the value of this parameter, the object is downloaded by using resumable download. Unit: bytes. Default value: 104857600. This value is equivalent to 100 MB. Valid values: 0 to 9223372036854775807. |
| --part-size | Specifies the part size. Unit: bytes. By default, ossutil determines the part size based on the object size. Valid values: 1 to 9223372036854775807. |
| --checkpoint-dir | Specifies the directory in which the log information about resumable download tasks is stored. If a resumable download task fails, ossutil automatically creates the .ossutil_checkpoint directory and stores the checkpoint information in the directory. ossutil deletes the .ossutil_checkpoint directory after the task is completed. Therefore, if you specify a checkpoint directory, make sure that the directory can be deleted. |
| --range | Specifies that a specific range of the object content is downloaded and stored as a new file in the destination local path. The minimum start value of the range is 0, which indicates the byte 0 of the content of the object. You can specify a value for this parameter in one of the following formats:
|
| --encoding-type | Specifies the method that is used to encode the names of objects. Set the value to url. If you do not specify this parameter, the names of objects are not encoded. |
| --include | Specifies that the command takes effect on all objects that meet the specified conditions. Note You cannot use this parameter to specify that the command takes effect on directories, such as ./ossutil64 cp oss://examplebucket/destfolder/ localfolder/ --include dir/. Otherwise, the following error is reported: --include or --exclude does not support format containing dir info. |
| --exclude | Specifies that the objects that meet the specified exclusion conditions are not downloaded. Note You cannot use this parameter to specify that the command takes effect on directories, such as ./ossutil64 cp oss://examplebucket/destfolder/ localfolder/ --exclude dir/. Otherwise, the following error is reported: --include or --exclude does not support format containing dir info. |
| --meta | Specifies the object metadata in the header:value#header:value format. Example: Cache-Control:no-cache#Content-Encoding:gzip. For more information about object metadata, see set-meta (manage object metadata). |
| --acl | The access control list (ACL) of the objects. Default value: private. Valid values:
|
| --snapshot-path | Specifies the directory in which the snapshots of downloaded objects are stored. In the next download task, ossutil reads the snapshots in this directory to download only incremental objects. |
| --disable-crc64 | Specifies that CRC-64 is disabled. By default, ossutil enables CRC-64 during data transmission. |
| --payer | Specifies the payer of the request. If you want the requester who accesses the resources in the specified path to pay the traffic and request fees, set this parameter to requester. |
| --partition-download | Specifies the partition that you want to download. The value of this parameter is in the partition number: the total number of partitions format. A value of 1:5 specifies that the object is split into 5 partitions and partition 1 is downloaded. Partitions are numbered starting from 1. Partitioning rules for objects are determined by ossutil. This option splits an object into multiple partitions that can be concurrently downloaded by multiple ossutil commands. Each ossutil command downloads the corresponding partition. |
| -j, --job | The number of concurrent tasks that can be performed across multiple objects. Valid values: 1 to 10000. Default value: 3. |
| --parallel | Specifies the number of concurrent tasks to download a single object. Valid values: 1 to 10000. By default, if you do not configure this parameter, ossutil specifies a value based on the operation type and the object size. |
| --version-id | Specifies the version ID of the object that you want to download. You can use this parameter for objects only in a bucket for which versioning is enabled. |
The -j, --jobs and --parallel parameters in the preceding command syntax can be used to adjust performance if the default number of concurrent tasks does not meet your performance requirements. By default, ossutil calculates the number of concurrent operations based on the object size. When you download multiple large objects, the actual number of concurrent tasks is the value of -j, --jobs multiplied by the value of --parallel.
- We recommend that you adjust the number of concurrent tasks to a value less than 100 if the resources such as the network bandwidth, memory, and CPU of your ECS instance or server are limited. If the usage of resources such as network bandwidth, memory, and CPU is low, you can increase the number of concurrent operations.
- If the number of concurrent tasks is too large, the download performance of ossutil may decrease, or an end-of-file (EOF) error may occur because thread resources are switched and threads compete for resources. To prevent this issue, you can adjust the values of the -j, --jobs and --parallel parameters based on the actual available resources. To perform stress testing, we recommend that you initially specify small values for the two parameters and gradually increase the values to find the optimal values.
Sample environment
In this topic, objects are downloaded from OSS to your local directory in Linux. You can change the parameter configurations based on your operating system and environment. Sample environment:
- Operating system: Linux
- Bucket: examplebucket
- Directory of the bucket: destfolder
- Local file: examplefile.txt. This file is stored in the root directory.
- Local directory: localfolder. This directory is stored in the root directory.
Download a single object
If you do not specify a name for the downloaded object, the name of the source object is used by default. If you specify a name for the downloaded object, the specified name is used in the local device.
- You can run the following command to save the downloaded object with the name of the source object:
./ossutil64 cp oss://examplebucket/destfolder/examplefile.txt localfolder/ - You can run the following command to save the downloaded object with a specified name:
./ossutil64 cp oss://examplebucket/destfolder/examplefile.txt localfolder/example.txt
Download multiple objects at a time
You cannot specify multiple object names in ossutil to download multiple objects at a time. You can download multiple objects at a time by using the following methods:
- Download objects of a specific type
If you want to download multiple objects whose names contain the same prefix or suffix, you can use the --include and --exclude parameters to download objects that meet specific conditions.
- You can run the following command to download all objects that are not in the JPG format:
./ossutil64 cp oss://examplebucket/destfolder/ localfolder/ --exclude "*.jpg" -r - You can run the following command to download all objects that contain abc in their names and are not in the JPG format or TXT format:
./ossutil64 cp oss://examplebucket/destfolder/ localfolder/ --include "*abc*" --exclude "*.jpg" --exclude "*.txt" -r
- You can run the following command to download all objects that are not in the JPG format:
- Download a directory and its subdirectories
You can run the following command to download all objects in a directory, including subdirectories in the directory:
./ossutil64 cp -r oss://examplebucket/destfolder/ localfolder/If a batch download task fails or you want to perform an incremental download task, you can use the --update parameter or its shortened form -u to skip objects that have been downloaded. If an object does not have the same name as local files, or the object is last modified later than the local file that has the same name, ossutil downloads the object. If an object has the same name as a local file, and the local file that has the same name is last modified later than the object, ossutil skips the object.
./ossutil64 cp -r oss://examplebucket/destfolder/ localfolder/ --update - Download a directory (excluding its subdirectories)
If you want to download only the current directory and do not want to download its subdirectories, use the --only-current-dir parameter.
./ossutil64 cp oss://examplebucket/destfolder/ localfolder/ --only-current-dir -r - Download all objects in the root directory of a bucket
Download all objects in the root directory of a bucket named examplebucket to the local directory named localfolder/.
./ossutil64 cp -r oss://examplebucket localfolder/
Limit download speed
- You can run the following command to download objects from OSS and set the --maxdownspeed parameter to 1 MB/s:
./ossutil64 cp oss://examplebucket/destfolder/examplefile.txt localfolder/ --maxdownspeed 1024 - You can run the following command to download directories from OSS and set the value of the --maxdownspeed parameter to 1 MB/s:
./ossutil64 cp -r oss://examplebucket/destfolder/ localfolder/ --maxdownspeed 1024
Download a range of content of an object
./ossutil64 cp oss://examplebucket/destfolder/examplefile.txt localfolder/ --range=10-20
Succeed: Total num: 1, size: 11. OK num: 1(download 1 objects).Download multiple objects at a time and generate snapshot information
./ossutil64 cp -r oss://examplebucket/destfolder/ localfolder/ --snapshot-path=path Download a specified version of an object from a versioning-enabled bucket
./ossutil64 cp oss://my-bucket/test.jpg localfolder/ --version-id CAEQARiBgID8rumR2hYiIGUyOTAyZGY2MzU5MjQ5ZjlhYzQzZjNlYTAyZDE3MDRk- To use the --version-id parameter, you must run the ls --all-versions command to obtain the version IDs of the object.
- The --version-id parameter can be specified in the command to download objects only from a versioning-enabled bucket. For more information about the command used to enable versioning for a bucket, see bucket-versioning.
Common options
If you use ossutil to switch to a bucket that is located in another region, add the -e option to the command to specify the endpoint of the region in which the specified bucket is located. If you use ossutil to switch to a bucket that belongs to another Alibaba Cloud account, you can add the -i option to the command to specify the AccessKey ID of the specified account, and add the -k option to the command to specify the AccessKey secret of the specified account.
./ossutil64 cp oss://examplebucket/exampleobject.txt localfolder/ -e oss-cn-shanghai.aliyuncs.com -i LTAI4Fw2NbDUCV8zYUzA**** -k 67DLVBkH7EamOjy2W5RVAHUY9H****For more information about other common options that you can specify in the sync command, see Common options.