Advanced commands - Object Storage Service - Alibaba Cloud Documentation Center

This topic describes the advanced commands that you can use.

Command descriptions

Command scope	Supported options
All advanced commands	--encoding-type string: The codec for the input object name or file name. Valid value: url. --request-payer string: The payment method for the request. If you use the pay-by-requester mode, set this parameter to `requester`.
Commands that support batch operations	--start-after/--end-with (, ]: Specifies the query range of keys. The range is exclusive of the start value and inclusive of the end value. filter options: You can set filter conditions based on information such as object/file names, object/file folders, object/file sizes, object/file times, and object metadata. For more information about the specific filtering rules, see Filter options. --limited-num: Sets the data returned by the query API. --recursive/-r: Performs a recursive operation. Recursively accesses all files or objects in the root directory, including subdirectories. --dirs/-d: Accesses only the files or objects in the root directory, not including subdirectories. Note For objects, this is impersonated using the Delimiter method. This requires scanning all objects under the prefix. The more objects there are, the more time-consuming the operation is. --force/-f: Forces the operation without a confirmation prompt. --list-objects: Uses the ListObjects API to list objects.
Commands that support destination filtering rules	--update: Compares only the modification time. Synchronizes only when the source is newer than the destination. --size-only: Compares only the file size. Synchronizes only data with a different file size. --checksum: Compares the CRC-64 checksum. The file size is compared first. If the file sizes are the same, the CRC-64 checksum is then compared. If the CRC-64 checksum does not exist on one side, the files are considered inconsistent. This is valid only for copying between objects. --ignore-existing: Skips existing files.
Commands that support single objects	--version-id string: The version ID of the object.
Commands that support list mode	--list-format: The format of the list file. Valid values: plain and inventory. --list-manifest-from: Reads the description of the list file format from a file. This parameter is required when the list file format is inventory.

Local options

Local option	Option type	Description
--encoding-type	string	The codec for the input object name or file name. Valid value: url.
--request-payer	string	The payment method for the request. If you use the pay-by-requester mode, set this parameter. Valid value: requester.
--start-after	string	Returns objects that are alphabetically after the specified value. The specified object is not included.
--end-with	string	Returns objects that are alphabetically before the specified value. The specified object is included.
--page-size	int	The maximum number of objects to list per page during batch processing. Default value: 1000. Value range: 1 to 1000.
--include	stringArray	The inclusion rule for paths or file names.
--include-from	stringArray	Reads inclusion rules from a rule file.
--exclude	stringArray	The exclusion rule for paths or file names.
--exclude-from	stringArray	Reads exclusion rules from a rule file.
--filter	stringArray	The filter rule for paths or file names.
--files-from	stringArray	Reads a list of source file names from a file. Empty lines or comment lines are ignored. This applies only to file filtering operations.
--files-from-raw	stringArray	Reads a list of source file names from a file. This applies only to file filtering operations.
--max-age	Duration	Does not transfer any file older than this age. The default unit is seconds. You can also use a unit suffix, such as ms, s, m, h, d, w, M, or y. Default value: off.
--min-age	Duration	Does not transfer any file newer than this age. The default unit is seconds. You can also use a unit suffix, such as ms, s, m, h, d, w, M, or y. Default value: off.
--max-mtime	Time	Does not transfer any file newer than this time. Use the UTC format. Default value: off.
--min-mtime	Time	Does not transfer any file older than this time. Use the UTC format. Default value: off.
--max-size	SizeSuffix	The maximum size of a file to transfer. The default unit is bytes. You can also use a unit suffix, such as B, K, M, G, T, or P. 1 K (KiB) = 1024 B.
--min-size	SizeSuffix	The minimum size of a file to transfer. The default unit is bytes. You can also use a unit suffix, such as B, K, M, G, T, or P. 1 K (KiB) = 1024 B.
--metadata-exclude	stringArray	The exclusion rule for object metadata.
--metadata-include	stringArray	The inclusion rule for object metadata.
--metadata-filter	stringArray	The filter rule for object metadata.
--metadata-filter-from	stringArray	Reads object metadata filter rules from a rule file.
--checksum	/	Copies only source files whose size and checksum (if it exists) are inconsistent.
--size-only	/	Copies only source files with an inconsistent size.
-u, --update	/	Copies only source files that are newer than the destination files.
-r, --recursive	/	Performs a recursive operation. When this option is specified, the command is performed on all objects in the bucket that meet the conditions. Otherwise, the command is performed only on the object specified by the path.
-d, --dirs	/	Returns the files and subdirectories in the current folder, instead of recursively displaying all files in all subdirectories.
-f, --force	/	Forces the operation without a confirmation prompt.
--list-objects	/	Uses the ListObjects API to list objects.
--no-progress	/	Does not display the progress bar.
-j, --job	int	The number of concurrent tasks for multi-file operations. Default value: 3.
--checkers	int	The number of checkers to run in parallel. Default value: 16.
--parallel	int	The number of concurrent tasks for operations within a single file.
--part-size	SizeSuffix	The part size. By default, an appropriate part size is calculated based on the file size. Value range: 100 KiB to 5 GiB.
--bigfile-threshold	int	The threshold to enable multipart upload, download, or copy for large files. Default value: 104857600.
--version-id	string	The version ID of the object.
--storage-class	string	The storage class of the object. Valid values: Standard, IA, Archive, ColdArchive, and DeepColdArchive.
--acl	string	The access permissions of the object. Valid values: private, public-read, public-read-write, and default.
--cache-control	string	Specifies the web page caching behavior when the object is downloaded.
--content-disposition	string	Specifies the display format of the object.
--content-encoding	string	Declares the codec of the object.
--content-type	string	The content type of the object.
--expires	string	Specifies the absolute expiration time of the cached content.
--copy-props	string	Determines which properties to copy from the source object. Valid values: none, metadata, and default.
--tagging	string	Specifies the tags of the object in the key=value format.
--tagging-directive	string	Specifies how to set the tags of the destination object. Valid values: COPY and REPLACE.
--metadata	string	Specifies the user metadata of the object in the key=value format.
--metadata-directive	string	Specifies how to set the metadata of the destination object. Valid values: COPY and REPLACE.
--list-format	string	The format of the list file. Valid values: plain and inventory.
--list-manifest-from	string	Reads the description of the list file format from a file. This parameter is required when the list file format is inventory.

Filter options

Filter options apply to commands that support batch operations, such as ls, cp, sync, and rm. These options allow you to filter items based on path or file names, path or file lists, file modification times, and file sizes. For objects, you can also filter by metadata.

You can use filter options only on the command line. You cannot specify them in the configuration file.

To test filter configurations, you can use the ls command for verification. You can also use the --dry-run/-n option to perform a trial run without making any changes.

Filter by path/file name

To match files by name or path, you must define filter rules that specify which files to include and which to exclude.

You can define rules using the --include and --exclude options. For example, --include="*.txt" and --exclude="*".

You can specify rules in command-line options or in a rule file. You can also combine and use multiple rules. The complete set of rules forms a list that filters results according to the following principles:

Rules are processed in order. Parameters in a command are processed from left to right, and rules in a file are processed from top to bottom.
Each file or path is checked against the rules to determine whether it should be included or excluded.
The rules are checked in order. The check stops as soon as a rule is matched.
If no rules are matched, the file or path is included by default.
On a local file system, if a folder is excluded, all its subfolders and files are also excluded.
Because buckets do not have a folder structure, each object is filtered individually.

Glob matching patterns are used. The following are common rules for glob matching patterns:

If a pattern starts with /, it must match the beginning of the path. Otherwise, it can match a name at any level of the path.
If a pattern ends with /, it matches only folders. Otherwise, it can match folders or files. For example, dir/.
A / character represents a path separator. For example, subdir/test.txt.
* matches any string of characters of any length, but does not match /. For example, *.txt.
If a pattern ends with **, it matches any string of characters of any length, including /. For example, dir/** matches the path "dir/subdir/xx/test.txt".
? matches any single character except /.
[] matches a single character from a specific class. For example, [a-z] matches a lowercase letter, and [0-9] matches a digit.
By default, a pattern must match the entire file or folder name. For example, `foo` matches "foo" and "dir/foo", but does not match "dir/foo1" or "foo/xx".
The pattern must match a complete folder or file name, not part of a name. For example, `foo` does not match "xxx/afoo", and `abc/foo` does not match "subdir/aabc/foo".

Note

On Windows, use / as the path separator, not \.

You can create an ordered list of path and file name filter rules by specifying one or more of the following syntax and parameters on the command line:

Item	Description
Filter rule syntax	rule pattern rule option parameter: A Glob matching pattern. rule filename rule option parameter: The path of a file that contains Glob matching pattern rules. rule modifiers pattern rule option parameter: An instruction Glob matching pattern. The `+` instruction indicates an inclusion rule. Objects that match this rule are included in the result. The `-` instruction indicates an exclusion rule. Objects that match this rule are excluded from the result.
Filter rule option parameters	--include --include-from --exclude --exclude-from --filter --filter-from

The following table describes the common options:

Option

Description

--include

--exclude

The parameter is followed by a matching pattern. Each option represents one rule. To use multiple rules, repeat these options.

For example, --include "*.txt" --include "*.js" --exclude "*.jpg" includes all .txt and .js files, but excludes all .jpg files.

The following example command recursively downloads files that match the filter rules from the oss://examplebucket/exampledir/ folder in Alibaba Cloud OSS to the localdir/ folder. The rules include all .txt and .js files, but exclude all .jpg files.

ossutil cp -r oss://examplebucket/exampledir/ localdir/ --include "*.txt" --include "*.js" --exclude "*.jpg"

--filter

The parameter is followed by an instruction matching pattern. Each option value represents one rule. To use multiple rules, repeat these options.

For example, --filter "+ *.txt" --filter "+ *.js" --filter "- *.jpg" includes all .txt and .js files, but excludes all .jpg files.

The + symbol indicates an inclusion rule. Objects that match this rule are included in the result.
The - symbol indicates an exclusion rule. Objects that match this rule are excluded from the result.

The following example command recursively downloads files that match the filter rules from the oss://examplebucket/exampledir/ folder in Alibaba Cloud OSS to the localdir/ folder. The rules exclude .txt and .js files, and include only .png files.

ossutil cp -r oss://examplebucket/exampledir/ localdir/ --filter "- *.txt" --filter "- *.js" --filter "+ *.png"

--include-from

--exclude-from

The parameter is followed by a rule file path. Each line in the rule file represents a Glob matching pattern. To import multiple rule files, repeat these options.

All rules imported with --include-from are inclusion rules. All rules imported with --exclude-from are exclusion rules.

For example, --include-from include-file.txt --exclude-from exclude-file.txt includes all .txt and .js files, but excludes all .jpg files.

The content of the rule file include-file.txt is as follows:

 *.txt
 *.js

The rule file include-file.txt contains the following:

 *.jpg

--filter-from

The parameter is followed by a rule file path. Each line in the rule file represents an instruction Glob matching pattern.

To import multiple rule files, repeat these options.

For example, in --filter-from filter-file.txt, the content of the rule file filter-file.txt is as follows:

 # a filter rule file
 + *.txt
 + *.js
 - **

Additional information:

When you read rules from a file, the rules are processed from top to bottom. Empty lines and comment lines that start with # or ; are ignored.
If the file path is -, the rules are read from standard input (stdin). You can specify this option only once.
If you use the --include or --include-from rule, a - ** rule is added to the end of the rule list.
If you use the --filter + pattern, a - ** rule is not added to the end of the rule list.

For example, to configure a rule that includes only files with the .jpg extension using --filter, you must specify `--filter "+ *.jpg" --filter "- **"`.

Note

You cannot use the --filter or --filter-from options in combination with the --include, --include-from, --exclude, or --exclude-from options.

Filter by path/file list

You can specify a list of paths or file names using command-line options. The supported options are as follows:

Option

Description

--files-from

Reads a list of paths/file names from a file. You can repeat this option to use multiple list files.

When processing each line in the list file, leading or trailing spaces are removed, and comment lines starting with # or ; are ignored.

--files-from-raw

Reads a list of paths/file names from a file. You can repeat this option to use multiple list files.

When processing each line in the list file, leading or trailing spaces and comment lines starting with # or ; are not ignored.

Note

If you specify a list of paths or files to match, other pattern matching rules are ignored.

Filter by file time

You can set time filter rules to transfer only files or objects last modified within a specific time range.

Time filter rules support two modes: relative time and absolute time. The configuration parameters are as follows:

Parameter		Description
--min-age	Duration	Processes only files modified before the specified time interval. The default unit is seconds. You can use a unit suffix. For example, 1h indicates 1 hour. Note `--min-age 1h` means process files modified 1 hour ago or earlier.
--max-age	Duration	Processes only files modified within the specified time interval. The default unit is seconds. You can use a unit suffix. For example, 1h indicates 1 hour. Note `--max-age 1h` means process files modified within the last hour.
--min-mtime	Time	Processes only files modified after the specified time. The time format is UTC. For example, 2006-01-02T15:04:05. Note `--min-mtime "2006-01-02T15:04:05"` means process files modified after 15:04:05 on January 2, 2006.
--max-mtime	Time	Processes only files modified before the specified time. The time format is UTC. For example, 2006-01-02T15:04:05. Note `--max-mtime "2006-01-02T15:04:05"` means process files modified before 15:04:05 on January 2, 2006.

Additional information:

The following unit suffixes are supported by --min-age and --max-age: ms for milliseconds, s for seconds, m for minutes, h for hours, d for days, w for weeks, M for months, and y for years.
When you use a unit suffix, you can use decimal values. For example, 1.5d represents one and a half days.

Example: List files or objects that are more than 3 days old.

ossutil ls oss://bucket/ --min-age 3d

Filter by file size

You can set file size-based filter rules to transfer only files or objects that are within a specific size range.

The configuration parameters for file size filter rules are as follows:

Parameter	Description
--min-size	Does not transfer any file smaller than this size. The default unit is bytes (B). You can also use a unit suffix. For example, 1K (KiB) represents 1024 bytes.
--max-size	Does not transfer any file larger than this size. The default unit is bytes (B). You can also use a unit suffix. For example, 1K (KiB) represents 1024 bytes.

Additional information:

The following unit suffixes are supported: K (KiB) = 1024 bytes, M (MiB), G (GiB), T (TiB), P (PiB), and E (EiB).
When you use a unit suffix, you can use decimal values. For example, 0.5K.

Example: List files or objects that are 1 MiB or smaller.

ossutil ls oss://bucket/ --max-size 1M

Filter by object metadata matching rules

Metadata filters are used in a similar way to path and file name filters, but they apply only to objects in a bucket.

In a metadata filter, you define the object metadata as `key=value` and can use glob patterns for matching.

The following configuration options are supported:

--metadata-include
--metadata-exclude
--metadata-filter
--metadata-filter-from

The following object metadata is supported:

Storage class: Use `x-oss-storage-class=value`. Valid values for `value` are Standard, IA, Archive, ColdArchive, and DeepColdArchive.
Object type: Use `x-oss-object-type=value`. Valid values for `value` are Normal, Multipart, Appendable, and Symlink.
Restore status: Use `x-oss-restore=value`. Set `value` as needed.
Content-Type: Use `content-type=value`. Set `value` as needed.
User-defined metadata: Use `x-oss-meta-aaa=value`, where `aaa` is in lowercase. Set `value` as needed.

Example: List objects whose storage class is Archive, Cold Archive, or Deep Cold Archive.

ossutil ls oss://bucket/ --metadata-include "x-oss-storage-class=*Archive"