All Products
Search
Document Center

Object Storage Service:Download objects

Last Updated:Nov 27, 2023

This topic describes how to run the cp command to download objects from Object Storage Service (OSS) to your on-premises device.

Important

Sample command lines in this topic are based on the 64-bit Linux system. For other systems, replace ./ossutil64 in the commands with the corresponding binary name. For more information, see ossutil.

Command syntax

./ossutil64 cp cloud_url file_url
[-r, --recursive]
[-f --force]
[-u --update]
[--maxdownspeed <value>]
[--disable-ignore-error]
[--only-current-dir]
[--bigfile-threshold <value>]
[--part-size <value>]
[--checkpoint-dir <value>]
[--range <value>]
[--encoding-type <value>]
[--include <value>]
[--exclude <value>]
[--meta <value>]
[--acl <value>]
[--snapshot-path <value>]
[--disable-crc64]
[--payer <value>]
[--partition-download <value>]
[-j, --job <value>]
[--parallel <value>]
[--version-id <value>]
[--start-time <value>]
[--end-time <value>]

The following table describes the parameters and options that you can configure to run the cp command.

Item

Description

cloud_url

The path of the object that you want to download from OSS. The path format is oss://bucketname/objectname. Example: oss://examplebucket/examplefile.txt.

file_url

The local path to which you want to download the object from OSS. An example file path on Linux is /localfolder/examplefile.txt. An example file path on Windows is D:\localfolder\examplefile.txt.

-r, --recursive

The recursive operations. If you configure this option, ossutil performs operations on all objects in a bucket that meet the specified condition. If you do not specify this option, ossutil performs operations only on the specified object.

-f --force

Forces an operation without requiring user confirmation.

-u, --update

Specifies that ossutil downloads an object from OSS only when the destination file does not exist in the local device or when the last modification time of the source object is later than that of the destination file.

--maxdownspeed

Specifies the maximum download speed. Unit: KB/s. The default value is 0, which indicates that the download speed is unlimited.

--disable-ignore-error

Specifies that errors are not ignored during batch operations.

--only-current-dir

Specifies that only objects in the current directory are downloaded. Subdirectories in the current directory and objects in these subdirectories are not downloaded.

--bigfile-threshold

Specifies the object size threshold for using resumable download. If an object exceeds the value of this option, the object is downloaded by using resumable download. Unit: bytes.

Default value: 104857600 (100 MB).

Valid values: 0 to 9223372036854775807.

--part-size

Specifies the part size. Unit: bytes. By default, ossutil determines the part size based on the object size.

Valid values: 1 to 9223372036854775807.

--checkpoint-dir

Specifies the directory in which the checkpoint information about resumable download is stored. If a resumable download task fails, ossutil automatically creates a directory named .ossutil_checkpoint and saves the checkpoint information in the directory. If the resumable download task is successful, ossutil deletes the directory. Therefore, if you specify a checkpoint directory, make sure that the directory can be deleted.

--range

Specifies that a specific range of the object content is downloaded and stored as a new file in the destination local path. The minimum start value of the range is 0, which indicates the byte 0 of the content of the object. You can specify a value for this option in one of the following formats:

  • Specifies the data range that you want to download.

    For example, a value of 3-9 indicates a range from byte 3 to byte 9.

  • Start value

    For example, a value of 3- indicates a range from byte 3 to the end of the object.

  • End value

    For example, the value of -9 indicates a range from byte 0 to byte 9.

--encoding-type

Specifies the method that is used to encode the names of objects. Set the value to url. If you do not specify this option, the names of objects are not encoded.

--include

Specifies that the objects that meet the specified inclusion conditions are downloaded.

For more information, see Options --include and --exclude.

--exclude

Specifies that the objects that meet the specified exclusion conditions are not downloaded.

For more information, see Options --include and --exclude.

--meta

Specifies object metadata in the header:value#header:value format. Example: Cache-Control:no-cache#Content-Encoding:gzip. For more information about object metadata, see set-meta (manage object metadata).

--acl

The access control list (ACL) of the objects. Valid values:

  • default: The ACL of the objects is the same as the ACL of the bucket in which the objects are stored.

  • private: Only the bucket owner can perform read and write operations on objects in the bucket. Other users cannot access the objects in the bucket.

  • public-read: Only the bucket owner can perform read and write operations on the objects in the bucket. Other users, including anonymous users, can perform only read operations on the objects in the bucket. This may result in unauthorized access to the data in your bucket and high fees. Proceed with caution when you set the object ACL to public-read.

  • public read-write: All users, including anonymous users, can perform read and write operations on the objects in the bucket. This may result in unauthorized access to the data in your bucket and high fees. If a user uploads prohibited data or information, your legitimate interests and rights may be infringed. Therefore, we recommend that you do not set the object ACL to public-read-write except in special cases.

--snapshot-path

Specifies the directory in which the snapshots of downloaded objects are stored. In the next download task, ossutil reads the snapshots in this directory to download only incremental objects.

--disable-crc64

Specifies that CRC-64 is disabled. By default, ossutil enables CRC-64 during data transmission.

--payer

Specifies the payer of the request. If you want the requester who accesses the resources in the specified path to pay the traffic and request fees, set this option to requester.

--partition-download

Specifies the partition in which the object that you want to download is stored. The value of this option is in the Number of partition to download:Total number of partitions format. For example, the value 1:5 specifies that the object is split into five partitions and the current command downloads partition 1. Partitions are numbered starting from 1. Partitioning rules for objects are determined by ossutil. This option splits an object into multiple partitions that can be concurrently downloaded by multiple ossutil commands. Each ossutil command downloads its own partition.

-j, --job

Specifies the number of concurrent tasks performed across multiple objects. Valid values: 1 to 10000. Default value: 3.

--parallel

Specifies the number of concurrent tasks to download a single object. Valid values: 1 to 10000. If you do not configure this option, ossutil specifies a value based on the operation type and the object size.

--version-id

Specifies the version ID of the object that you want to download. You can use this option for objects only in a bucket for which versioning is enabled.

--start-time

Specifies the start of the time range to filter objects by last modified time. The value is a UNIX timestamp. Objects whose last modified time is earlier than the UNIX timestamp are ignored.

--end-time

Specifies the end of the time range to filter objects by last modified time. The value is a UNIX timestamp. Objects whose last modified time is later than the timestamp are ignored.

Note

If both --start-time and --end-time are specified, the command applies only to objects whose last modified time is within the range specified by --start-time and --end-time.

The -j, --jobs and --parallel options in the preceding command syntax can be used to adjust performance if the default number of concurrent tasks does not meet your performance requirements. By default, ossutil calculates the number of concurrent operations based on the object size. When you download multiple large objects, the actual number of concurrent tasks is the value of -j, --jobs multiplied by the value of --parallel.

  • We recommend that you adjust the number of concurrent tasks to a value smaller than 100 if your Elastic Compute Service (ECS) instance or server has limited resources such as network bandwidth, memory, and CPU. If the usage of resources such as network bandwidth, memory, and CPU is low, you can increase the number of concurrent operations.

  • If the number of concurrent tasks is too large, the download performance of ossutil may decrease because thread resources are switched and threads compete for resources. In addition, an end-of-file (EOF) error may occur. To resolve this issue, you must adjust the values of the -j, --jobs and --parallel options based on the actual available resources. To perform stress testing, we recommend that you initially specify small values for the two options and gradually increase the values to find the optimal values.

Sample environment

In this topic, objects are downloaded from OSS to your local directory in Linux. You can change the option configurations based on your operating system and environment. Sample environment:

  • Operating system: Linux

  • Bucket: examplebucket

  • Directory of the bucket: destfolder

  • Local file: examplefile.txt (a file in the root directory)

  • Local directory: localfolder (a directory in the root directory)

Download a single object

If you do not specify a name for the downloaded object, the name of the downloaded object is the same as the original object. If you specify a name for the downloaded object, the downloaded object uses the specified name.

  • Run the following command to save the downloaded object with the name of the original object:

    ./ossutil64 cp oss://examplebucket/destfolder/examplefile.txt localfolder/
  • Run the following command to save the downloaded object with a specified name:

    ./ossutil64 cp oss://examplebucket/destfolder/examplefile.txt localfolder/example.txt

Download multiple objects at a time

You cannot specify multiple object names in ossutil to download multiple objects at a time. You can download multiple objects at a time by using the following methods:

  • Download objects of a specific type

    If you want to download multiple objects whose names contain the same prefix or suffix, you can use the --include and --exclude options to download objects that meet specific conditions.

    • You can run the following command to download all objects that are not in the JPG format:

      ./ossutil64 cp oss://examplebucket/destfolder/ localfolder/ --exclude "*.jpg" -r
    • You can run the following command to download all objects that contain abc in their names and are not in the JPG format or TXT format:

      ./ossutil64 cp oss://examplebucket/destfolder/ localfolder/ --include "*abc*" --exclude "*.jpg" --exclude "*.txt" -r
  • Download a directory and its subdirectories

    You can run the following command to download all objects in a directory, including subdirectories in the directory:

    ./ossutil64 cp -r oss://examplebucket/destfolder/ localfolder/

    If a batch download task fails or you want to perform an incremental download task, you can use the --update option or its shortened form -u to skip objects that have been downloaded. If an object does not have the same name as a local file, or the object is last modified later than the local file that has the same name, ossutil downloads the object. If an object has the same name as a local file, and the local file that has the same name is last modified later than the object, ossutil skips the object. Sample command:

    ./ossutil64 cp -r oss://examplebucket/destfolder/  localfolder/  --update                           
  • Download a directory (excluding its subdirectories)

    If you want to download only the current directory and do not want to download its subdirectories, use the --only-current-dir option. Sample command:

    ./ossutil64 cp oss://examplebucket/destfolder/ localfolder/ --only-current-dir -r
  • Download a directory (with a timestamp specified)

    The following sample command downloads objects whose last modified time is from 10:09:18 (UTC+8) October 31, 2023 to 12:55:58 (UTC+8) October 31, 2023 from a directory and its subdirectories:

    ./ossutil64 cp -r oss://examplebucket/destfolder/ localfolder/ --start-time 1698718158 --end-time 1698728158
  • Download all objects in the root directory of a bucket

    The following sample command downloads all objects in the root directory of a bucket named examplebucket to the local directory named localfolder/:

    ./ossutil64 cp -r oss://examplebucket  localfolder/

Limit download speed

To limit the maximum download speed, you can specify the --maxdownspeed option. Unit: KB/s.

  • The following sample command downloads objects from OSS, with the maximum download speed limited to 1 MB/s:

    ./ossutil64 cp oss://examplebucket/destfolder/examplefile.txt  localfolder/ --maxdownspeed 1024
  • The following sample command downloads directories from OSS, with the maximum download speed limited to 1 MB/s:

    ./ossutil64 cp -r  oss://examplebucket/destfolder/  localfolder/ --maxdownspeed 1024

Download a range of the content of an object

To download a range of the content of an object, specify the --range option. For example, you can run the following command to download the 10th to 20th characters of the examplefile.txt object to your local device as a file:

./ossutil64 cp oss://examplebucket/destfolder/examplefile.txt localfolder/  --range=10-20
Succeed: Total num: 1, size: 11. OK num: 1(download 1 objects).

Download multiple objects at a time and generate snapshot information

If you specify the --snapshot-path option when you download multiple objects at a time, ossutil generates a snapshot for the objects and stores the snapshot information in a specified directory. If you specify this option, ossutil reads the snapshot information from the specified directory and downloads only incremental objects next time you download the objects. For more information, see Generate snapshots for objects when you upload the objects.

./ossutil64 cp -r oss://examplebucket/destfolder/ localfolder/ --snapshot-path=path                                

Download a specified version of an object from a versioning-enabled bucket

After you enable versioning for a bucket, objects that are overwritten or deleted in the bucket are saved as previous versions. You can specify the --version-id option in the cp command to download the specified version of the object.

./ossutil64 cp oss://my-bucket/test.jpg localfolder/ --version-id  CAEQARiBgID8rumR2hYiIGUyOTAyZGY2MzU5MjQ5ZjlhYzQzZjNlYTAyZDE3MDRk
Important
  • To use the --version-id option, you need to run the ls --all-versions command to obtain the version IDs of the object.

  • The --version-id option can be used only for objects in versioning-enabled buckets. For more information about how to enable versioning for a bucket, see bucket-versioning.

Common options

If you use ossutil to switch to a bucket that is located in another region, add the -e option to the command to specify the endpoint of the region in which the specified bucket is located. If you use ossutil to switch to a bucket that belongs to another Alibaba Cloud account, you can add the -i option to the command to specify the AccessKey ID of the specified account, and add the -k option to the command to specify the AccessKey secret of the specified account.

Note

The endpoint that you specify must correspond to the region in which the bucket is located. For more information, see Regions and endpoints.

For example, you can run the following command to download the exampleobject.txt object in the root directory of the examplebucket bucket, which is owned by another Alibaba Cloud account and located in the China (Shanghai) region, to the local directory named localfolder:

./ossutil64 cp oss://examplebucket/exampleobject.txt  localfolder/  -e oss-cn-shanghai.aliyuncs.com -i LTAI4Fw2NbDUCV8zYUzA****  -k 67DLVBkH7EamOjy2W5RVAHUY9H****

For more information about other common options that you can use for the sync command, see View options.