All Products
Search
Document Center

Object Storage Service:Upload objects

Last Updated:Mar 18, 2024

You can run the cp command to upload resources such as local files, images, and videos to Object Storage Service (OSS). ossutil is particularly suitable for uploading large files. You can also include --include and --exclude options in the command to batch upload files that meet the specified conditions.

Usage notes

  • Sample command lines in this topic are based on the 64-bit Linux system. For other systems, replace ./ossutil64 in the commands with the corresponding binary name. For more information, see ossutil command reference.

  • By default, when you run the cp command to upload an object, multipart upload and resumable upload are used. If the upload is interrupted, the uploaded data is stored as parts in the OSS bucket. If you no longer need the uploaded parts, we recommend that you delete them by using one of the following methods to avoid additional storage costs:

    • Manually delete parts. For more information, see rm.

    • Configure lifecycle rules to automatically delete parts. For more information, see Configure lifecycle rules.

Command syntax

./ossutil64 cp file_url cloud_url
[-r, --recursive]
[-f --force]
[-u --update]
[--maxupspeed <value>]
[--enable-symlink-dir]
[--disable-all-symlink]
[--disable-ignore-error]
[--only-current-dir]
[--bigfile-threshold <value>]
[--part-size <value>]
[--checkpoint-dir <value>]
[--encoding-type <value>]
[--include <value>]
[--exclude <value>]
[--meta <value>]
[--acl <value>]
[--snapshot-path <value>]
[--disable-crc64]
[--disable-dir-object]
[--payer <value>]
[--tagging <value>]
[-j, --job <value>]
[--parallel <value>]
[--start-time <value>]
[--end-time <value>]

The following table describes the parameters and options in the syntax.

Parameter/Option

Description

file_url

The path of the local file. For example, the path of the local file on Linux is /localfolder/examplefile.txt, and the path of the local file on Windows is D:\localfolder\examplefile.txt.

cloud_url

The path of the OSS object. Specify the path in the oss://bucketname/objectname format. Example: oss://examplebucket/examplefile.txt.

-r, --recursive

Specifies recursion. If you specify this option, ossutil performs the operation on all objects in the bucket that meet the specified condition. If you do not specify this option, ossutil performs the operation only on the specified object.

-f --force

Forces the operation without requiring confirmation.

-u, --update

Uploads files only when the destination objects do not exist or when the last modified time of the source files is later than that of the destination objects.

--maxupspeed

The maximum upload speed. Unit: KB/s. Default value: 0. A value of 0 specifies that the upload speed is not limited.

--enable-symlink-dir

Uploads the subdirectory to which the symbolic link points. By default, subdirectories are not uploaded.

--disable-all-symlink

Ignores the subdirectory to which the symbolic link points and all objects in this subdirectory during the upload.

--disable-ignore-error

Specifies that errors are not ignored during batch operations.

--only-current-dir

Uploads only objects in the current directory. Subdirectories in this directory and objects in the subdirectories are ignored.

--bigfile-threshold

The threshold for performing a resumable upload. If the object size exceeds the threshold, resumable upload is used. Unit: bytes.

The default object size threshold is 100 MB.

Valid values: 0 to 9223372036854775807.

--part-size

The part size. Unit: bytes. By default, ossutil determines the part size based on the object size.

Valid values: 1 to 9223372036854775807.

--checkpoint-dir

The directory in which information about resumable upload task is stored. If a resumable upload fails, ossutil automatically creates a directory named .ossutil_checkpoint and saves the checkpoint information in the directory. After the resumable upload is successful, ossutil deletes the directory. If you specify a checkpoint directory, make sure that the directory can be deleted.

--encoding-type

The method that is used to encode the names of objects. If you specify this option, set the value to url. If you do not specify this option, the names of objects are not encoded.

--include

Includes all objects that meet the specified conditions. For more information about the syntax and examples, see Batch upload objects that meet specified conditions.

--exclude

Excludes all objects that meet the specified conditions. For more information about the syntax and examples, see Batch upload objects that meet specified conditions.

--meta

The object metadata. Object metadata includes standard HTTP headers and user metadata. User metadata starts with x-oss-meta-. Specify the object metadata in the header:value#header:value format. Example: Cache-Control:no-cache#Content-Encoding:gzip. For more information about OSS metadata, see Manage object metadata.

--acl

The access control list (ACL) of the objects. Valid values:

  • default: The ACL of the objects is the same as the ACL of the bucket in which the objects are stored. This is the default ACL setting for objects.

  • private: Only the bucket owner can perform read and write operations on the objects in the bucket. Other users cannot access the objects in the bucket.

  • public-read: Only the bucket owner can perform write operations on the objects in the bucket. Other users, including anonymous users, can perform only read operations on the objects in the bucket. This may result in unrestricted access to the data in your bucket and unexpectedly high fees. If a user uploads prohibited data or information, your legitimate interests and rights may be infringed. We recommend that you do not set the object ACL to this value except in special cases.

  • public-read-write: All users, including anonymous users, can perform read and write operations on the objects in the bucket. This may result in unrestricted access to the data in your bucket and unexpectedly high fees. Exercise caution when you set the object ACL to public-read-write.

--snapshot-path

The directory in which the snapshots of uploaded objects are stored. In the next upload, ossutil reads the snapshot information from the specified directory and performs an incremental upload.

--disable-crc64

Disables 64-bit cyclic redundancy check (CRC-64). By default, ossutil enables CRC-64 during data transmission.

--disable-dir-object

Generates no object for the directory during object uploads.

--payer

The payer of the request. If you want the requester who accesses the resources in the specified path to pay for the traffic and request, set this option to requester.

--tagging

The tags of objects. Specify the tags in the TagkeyA=TagvalueA&TagkeyB=TagvalueB.... format.

-j, --jobs

The number of concurrent tasks performed across multiple objects. Valid values: 1 to 10000. Default value: 3.

--parallel

The number of concurrent tasks performed on a single object. Valid values: 1 to 10000. If you do not configure this option, ossutil specifies a value based on the operation type and the object size.

--start-time

The beginning of the time range to filter objects by last modified time. The value is a UNIX timestamp. Objects whose last modified time is earlier than the UNIX timestamp are ignored.

Note

Only ossutil V1.7.18 and later support this parameter. For more information about how to update ossutil, see update.

--end-time

The end of the time range to filter objects by last modified time. The value is a UNIX timestamp. Objects whose last modified time is later than the UNIX timestamp are ignored.

Note
  • If both --start-time and --end-time are specified, the command applies only to objects whose last modified time is within the range specified by --start-time and --end-time.

  • Only ossutil V1.7.18 and later support this parameter. For more information about how to update ossutil, see update.

You can use the -j, --jobs and --parallel options to adjust performance if the default concurrency value does not meet your performance requirements. By default, ossutil calculates the value of --parallel based on the object size. When you batch upload large objects, the actual concurrency value equals to the product of the --jobs value and the --parallel value.

  • We recommend that you adjust the concurrency value to a value smaller than 100 if the resources such as network bandwidth, memory, and CPU of your ECS instance or server are limited. If resources such as network bandwidth, memory, and CPU are left idle, you can increase the concurrency value.

  • If the number of concurrent tasks is too large, the upload performance of ossutil may decrease because threads switch and compete for resources. In addition, an end-of-file (EOF) error may occur. To prevent the issues, you can adjust the values of the -j, --jobs and --parallel options based on your available resources. To perform stress testing, specify small values for the two options before you incrementally increase them to the optimal values.

Sample environment

In this topic, local files or directories are uploaded from a Linux OS to OSS. You can modify the options in the examples based on your OS and environment. This topic uses the following sample resources:

  • Local file: examplefile.txt (a file in the root directory)

  • Local directory: localfolder (a subdirectory in the root directory)

  • Destination OSS bucket: examplebucket

  • Directory in the destination bucket: desfolder

Simple upload

If you want to upload resources, such as local files, images, and videos, to OSS, you can use ossutil. The following sample commands provide examples on how to use ossutil to upload data to OSS.

  • Upload a single local file

    If you do not specify the name for the uploaded object, the object in OSS uses the same name as the local file. If you specify the name for the uploaded object, the object in OSS uses the specified name.

    ./ossutil64 cp examplefile.txt oss://examplebucket/desfolder/
  • Upload files from a local directory

    You can add the -r option to the cp command to upload files from a local directory to an OSS path.

    ./ossutil64 cp -r localfolder/ oss://examplebucket/desfolder/
  • Upload files whose last modified time is within the specified time range from a local directory

    You can run the following command to upload files whose last modified time is within the range of 10:09:18 (UTC+8) October 31, 2023 to 12:55:58 (UTC+8) October 31, 2023 from a local directory to an OSS path.

    ./ossutil64 cp -r localfolder/ oss://examplebucket/desfolder/ --start-time 1698718158 --end-time 1698728158
  • Upload a local directory and the files inside

    You can add the -r option to the cp command and include the name of a local directory in the OSS path to upload the local directory and the files inside to an OSS path. For the specified directory, OSS does not create a directory object. For a subdirectory in the specified directory, OSS creates a zero-byte object whose name ends with a forward slash (/). If you want to create a directory object for the local directory, run the mkdir command.

    ./ossutil64 cp -r localfolder/ oss://examplebucket/desfolder/localfolder/
  • Upload a single file and specify --meta

    You can specify object metadata in the header:value#header:value... format by using the --meta option when you run the cp command to upload a file.

    ./ossutil64 cp examplefile.txt oss://examplebucket/desfolder/examplefile.txt --meta=Cache-Control:no-cache#Content-Encoding:gzip
  • Upload a directory and skip existing objects

    When you resume a failed batch upload task, you can specify the --update (abbreviated as -u) option to skip uploaded objects and perform an incremental upload. Sample command:

    ./ossutil64 cp -r localfolder/ oss://examplebucket/desfolder/ -u
  • Upload a file to a bucket for which pay-by-requester is enabled

    ./ossutil64 cp localfolder/examplefile.txt oss://examplebucket/ --payer=requester
  • Upload files only in the current directory

    ./ossutil64 cp localfolder/ oss://examplebucket/desfolder/ --only-current-dir -r
  • Upload a directory without generating an object for the uploaded directory

    In OSS, a directory is a zero-byte object and has a name that ends with a forward slash (/). If you specify the --disable-dir-object option in the cp command to upload a directory, OSS does not generate a directory object for the directory. However, you can view the directory in the OSS console. If you delete all objects in the directory, the directory also disappears.

    ./ossutil64 cp localfolder/ oss://examplebucket/desfolder/ --disable-dir-object -r
  • Upload objects in the subdirectory to which a symbolic link points

    ./ossutil64 cp localfolder/ oss://examplebucket/desfolder/ --enable-symlink-dir -r
  • Ignore the subdirectory to which a symbolic link points and all objects in this subdirectory during the upload

    ./ossutil64 cp localfolder/ oss://examplebucket/desfolder/ -r --disable-all-symlink

Configure the maximum upload speed

When you upload an object, you can set --maxupspeed to specify the maximum upload speed. Unit: KB/s. Sample command:

  • Upload an object to OSS and set the maximum upload speed to 1 MB/s

    ./ossutil64 cp examplefile.txt oss://examplebucket/desfolder/ --maxupspeed 1024
  • Upload a directory to OSS and set the maximum upload speed to 1 MB/s

    ./ossutil64 cp -r localfolder/ oss://examplebucket/desfolder/ --maxupspeed 1024

Upload an object and configure tags for the object

When you run the cp command to upload an object, you can specify the --tagging option to configure tags for the object. If you configure multiple tags for the object, separate them with ampersands (&). Sample command:

./ossutil64 cp examplefile.txt oss://examplebucket/desfolder/ --tagging "abc=1&bcd=2&……"

For more information, see object-tagging (add, modify, query, and delete object tags).

Upload an object and specify a storage class for the object

When you run the cp command to upload a local file, you can specify the --meta option to specify the storage class of the object. ossutil supports the following storage classes:

  • Standard: Standard

  • IA: Infrequent Access (IA)

  • Archive: Archive

If you do not specify a storage class for the object in the command, OSS applies the storage class of the bucket to the object. For more information, see Overview.

Sample commands:

  • Upload an object and set the storage class of the object to IA

    ./ossutil64 cp examplefile.txt oss://examplebucket/desfolder/ --meta X-oss-Storage-Class:IA
  • Upload a directory and set the storage class of all objects in the directory to Standard

    ./ossutil64 cp localfolder/ oss://examplebucket/desfolder/ --meta X-oss-Storage-Class:Standard -r

Upload an object and specify the ACL of the object

When you run the cp command to upload an object, you can specify the --acl option to set the ACL of the object. ossutil supports the following ACLs:

  • default

  • private

  • public-read

  • public-read-write

Sample command:

  • Upload an object and set the ACL of the object to private

    ./ossutil64 cp examplefile.txt oss://examplebucket/desfolder/ --acl private
  • Upload an object and set the ACL of the object to public-read

    ./ossutil64 cp localfolder/ oss://examplebucket/desfolder/ --acl public-read  -r

Upload an object and specify an encryption method for the object

When you upload an object, you can specify the server-side encryption method for the object to store the object in the encrypted state. Sample command:

  • Specify SSE-OSS AES-256 encryption when you upload an object

    ./ossutil64 cp examplefile.txt oss://examplebucket/desfolder/ --meta=x-oss-server-side-encryption:AES256
  • Specify SSE-KMS encryption without specifying a customer master key (CMK) ID when you upload an object

    ./ossutil64 cp examplefile.txt oss://examplebucket/desfolder/ --meta=x-oss-server-side-encryption:KMS

    When you set the encryption method to SSE-KMS, you are charged for the usage of keys. For more information, see Billing of KMS.

  • Specify SSE-KMS encryption with the specified CMK ID when you upload an object

    ./ossutil64 cp examplefile.txt oss://examplebucket/desfolder/ --meta=x-oss-server-side-encryption:KMS#x-oss-server-side-encryption-key-id:7bd6e2fe-cd0e-483e-acb0-f4b9e1******

For more information about server-side encryption, see Server-side encryption.

Generate snapshots for objects when you upload the objects

When you run the cp command to upload multiple objects at a time and specify --snapshot-path in the command, ossutil generates the snapshots for the uploaded objects in the specified directory to record the last modified time of the objects. In subsequent upload tasks, ossutil determines whether to skip existing objects based on their last modified time. If you want to specify --snapshot-path in the command, make sure that the objects in OSS are not modified by other users since the last time when the objects were uploaded. The --snapshot-path option is used to accelerate incremental uploads. Sample command:

./ossutil64 cp -r localfolder/ oss://examplebucket/desfolder/ --snapshot-path=path                                
Important
  • ossutil does not automatically delete snapshots stored in the directory specified by snapshot-path. To prevent snapshots from consuming excessive storage space, you must delete the snapshots that you no longer need from the directory specified by snapshot-path.

  • Additional overheads are required to read and write snapshot information. We recommend that you do not use this option in the following scenarios: you only have a small number of objects to upload, the network condition is good, or other users need to perform operations on the objects. Instead, you can use --update to perform incremental uploads.

  • You can configure both --update and --snapshot-path in a cp command. ossutil determines whether to skip an object first based on the snapshots stored in the directory specified by --snapshot-path. If no snapshots are generated for the object, ossutil determines whether to skip the object based on the value of --update.

Batch upload objects that meet specified conditions

When you run the cp command to upload multiple objects at a time and specify the --include and --exclude options in the command, ossutil uploads objects that meet the specified conditions.

The --include and --exclude options support the following formats:

  • Asterisk (*): matches all characters. For example, *.txt matches all TXT files.

  • Question mark (?): matches a single character. For example, abc?.jpg matches all JPG objects whose names are abc followed by a single character, as in abc1.jpg.

  • [sequence]: matches characters in a sequence. For example, abc[1-5].jpg matches JPG objects whose names are abc followed by a number in sequence [1-5]. The object names are abc1.jpg, abc2.jpg, abc3.jpg, abc4.jpg, and abc5.jpg.

  • [!sequence]: matches characters outside a sequence. For example, abc[!0-7].jpg matches objects other than abc0.jpg, abc1.jpg, abc2.jpg, abc3.jpg, abc4.jpg, abc5.jpg, abc6.jpg, and abc7.jpg.

A rule can contain multiple conditions specified by --include and --exclude. After these conditions are configured, ossutil evaluates the conditions from left to right to obtain the final results. If the test.txt object exists in a directory for which conditions are specified, ossutil generates results based on different matching rules.

  • Rule 1: --include "*test*" --exclude "*.txt". ossutil first evaluates the --include "*test*" condition and includes the test.txt object in the results because the object is a match. Then, ossutil evaluates the --exclude "*.txt" condition, finds that the test.txt object is a match because the object name contains ".txt", and excludes the object from the results. The test.txt object is excluded from the final results.

  • Rule 2: --exclude "*.txt" --include "*test*". ossutil first evaluates the --exclude "*.txt" condition and excludes the test.txt object from the results because the object is a match. Then, ossutil evaluates the--include "*test*" condition, finds that the test.txt object is a match because the object name contains "test", and includes the object in the results. The test.txt object is included in the final results.

  • Rule 3: --include "*test*" --exclude "*.txt" --include "te?t.txt". ossutil first evaluates the --include "*test*" condition, finds that the test.txt object is a match, and includes the object in the results. Then, ossutil evaluates the --exclude "*.txt" condition, finds that the test.txt object is a match because the object name contains "txt", and excludes the object from the results. Finally, ossutil evaluates the --include "te?t.txt" condition, finds that the test.txt object is a match, and includes the object in the results. The test.txt object is included in the final results.

Important

You cannot specify directories in --include and --exclude conditions. For example, --include "/usr/test/.jpg" is invalid.

Sample commands:

  • Upload all TXT objects from the specified directory

    ./ossutil64 cp localfolder/ oss://examplebucket/desfolder/ --include "*.txt" -r
  • Upload all objects whose names contain abc and whose formats are not JPG or TXT from the specified directory

    ./ossutil64 cp localfolder/ oss://examplebucket/desfolder/ --include "*abc*" --exclude "*.jpg" --exclude "*.txt" -r

Common options

If you use ossutil to switch to a bucket that is located in another region, add the -e option to the command to specify the endpoint of the region in which the specified bucket is located. If you use ossutil to switch to a bucket that belongs to another Alibaba Cloud account, you can add the -i option to the command to specify the AccessKey ID of the specified account, and add the -k option to the command to specify the AccessKey secret of the specified account.

For example, you can run the following command to upload the local file exampleobject.txt to the destfolder directory of the examplebucket bucket that is located in the China (Shanghai) region and owned by another Alibaba Cloud account.

./ossutil64 cp exampleobject.txt oss://examplebucket/desfolder/ -e oss-cn-shanghai.aliyuncs.com -i LTAI4Fw2NbDUCV8zYUzA****  -k 67DLVBkH7EamOjy2W5RVAHUY9H****

For more information about common options, see Common options.