All Products
Search
Document Center

Object Storage Service:Synchronize OSS objects to your computer

Last Updated:Oct 08, 2023

You can run the sync command to synchronize objects that are stored in Object Storage Service (OSS) to your computer.

Usage notes

  • Binary name

    This topic provides sample command lines that are based on the 64-bit Linux system. For other systems, replace ./ossutil64 in the commands with the specific binary name. For more information, see ossutil.

  • Number of local files to synchronize

    When you run the sync command without including the --delete option, the number of local files that you can synchronize at a time is unlimited. If you include the --delete option, you can synchronize up to 1 million local files at a time. If the number of local files that you want to synchronize exceeds 1 million, the over max sync numbers 1000000. error message is returned.

  • Differences between the sync and cp commands

    • The sync command recursively synchronizes all objects and subdirectories in a specific directory. The cp command recursively synchronizes objects only if the -r option is specified.

    • The sync command supports the --backup-dir option. When you run the sync command to synchronize data to OSS, you can specify this option in the command to specify a directory in OSS to store objects that exist in the destination but do not exist in the source. The cp command does not support the --backup-dir option.

    • The sync command does not support the --version-id option. Therefore, the sync command cannot be used to synchronize previous versions of objects in versioned buckets. The cp command supports the --version-id option.

    Except for the preceding differences, you can use the sync and cp commands in a similar manner. For more information about how to run the cp command, see Download objects.

Command syntax

./ossutil64 sync cloud_url  file_url
[-f --force]
[-u --update]
[--maxdownspeed <value>]
[--delete]
[--backup-dir <value>]
[--enable-symlink-dir]
[--disable-all-symlink]
[--disable-ignore-error]
[--only-current-dir]
[--output-dir <value>]
[--bigfile-threshold <value>]
[--part-size <value>]
[--checkpoint-dir <value>]
[--range <value>]
[--encoding-type <value>]
[--snapshot-path <value>]
[--include <value>]
[--exclude <value>]
[--disable-crc64]
[--payer <value>]
[-j, --job <value>]
[--parallel <value>]
[--retry-times <value>]

The following table describes the options that you can configure to synchronize OSS objects to your computer by running the sync command.

Option

Description

cloud_url

Specifies the path of the directory in which the objects that you want to synchronize are stored. The path is in the oss://bucketname/path format. Example: oss://examplebucket/exampledir/. If the value of cloud_url does not end with a forward slash (/), ossutil automatically adds one to the end of the value.

file_url

Specifies the path of the local directory to which you want to synchronize objects. Examples: /localfolder/ in Linux and D:\localfolder\ in Windows.

-f --force

Specifies that the operation is forcibly performed. The operation is performed without a prompt for confirmation.

-u, --update

Specifies that ossutil synchronizes objects from the source only when the objects do not exist in the destination or when the last modified time of the object is later than that of the corresponding objects in the destination.

--maxdownspeed

Specifies the maximum download speed. Unit: KB/s. Default value: 0, which indicates that the download speed is unlimited.

--delete

Specifies that only the synchronized objects are retained in the destination local path. Other objects in the destination local path are deleted.

Warning

To specify the --delete option in the command, we recommend that you enable versioning for the bucket to prevent data from being accidentally deleted. For more information about versioning, see Overview.

--backup-dir

Specifies the directory used to store objects that exist in the destination local path but do not exist in the source OSS bucket.

--enable-symlink-dir

Specifies whether to synchronize subdirectories to which symbolic links point.

--disable-all-symlink

Specifies that all objects and subdirectories to which symbolic links in the directory point are not synchronized.

--disable-ignore-error

Specifies that errors are not ignored during batch operations.

--only-current-dir

Specifies that only objects in the current directory are synchronized. Subdirectories in the current directory and objects in these subdirectories are not synchronized.

--output-dir

Specifies the directory in which output objects are stored. The output objects are the report files generated when an error occurs in a batch synchronization task.

If you do not specify a directory for the output objects, the output objects are stored in the ossutil_output subdirectory of the current directory.

-bigfile-threshold

The object size threshold for using resumable download. Unit: bytes.

Default value: 104857600 (100 MB).

Valid values: 0 to 9223372036854775807.

--part-size

The part size. Unit: bytes. By default, ossutil determines the part size based on the object size.

Valid values: 1 to 9223372036854775807.

--checkpoint-dir

The directory in which the checkpoint information about resumable download tasks is stored. If a resumable download task fails, ossutil automatically creates a directory named .ossutil_checkpoint and saves the checkpoint information in the directory. If the resumable download task is successful, ossutil deletes the directory. Therefore, if you specify a checkpoint directory, make sure that the directory can be deleted.

--range

Specifies that a specific range of the object content is downloaded and stored as a new object in the destination local path. The minimum start value of the range is 0, which indicates the byte 0 of the content of the object. You can specify a value for this option in one of the following formats:

  • Data range

    For example, a value of 3-9 indicates a range from byte 3 to byte 9, which includes byte 3 and byte 9.

  • Start value

    For example, a value of 3- indicates a range from byte 3 to the end of the object, which includes byte 3.

  • End value

    For example, the value of -9 indicates a range from byte 0 to byte 9, which includes byte 9.

--encoding-type

Specifies the method that is used to encode the names of objects. Set the value to url. If you do not specify this option, the names of the objects are not encoded.

--snapshot-path

Specifies the directory in which the snapshots of synchronized objects are stored. In the next synchronization task, ossutil reads the snapshots in this directory for incremental synchronization.

--include

Specifies that the command takes effect on all objects that meet the specified conditions.

For more information, see Synchronize OSS objects to your computer.

--exclude

Specifies that the command takes effect on all objects that do not meet the specified conditions.

For more information, see Synchronize OSS objects to your computer.

--disable-crc64

Specifies that CRC-64 is disabled.

--payer

Specifies the payer of the request. If you want the requester who accesses the resources in the specified path to pay for the traffic and request fees, set this option to requester.

-j, --job

Specifies the number of concurrent tasks performed across multiple objects. Valid values: 1 to 10000. Default value: 3.

--parallel

Specifies the number of concurrent tasks performed on a single object. Valid values: 1 to 10000. If you do not specify this option, ossutil specifies the value of this option based on the operation type and the object size.

--retry-times

Specifies the number of retries after the command fails to be run. Default value: 10. Valid values: 1 to 500.

Examples

The localdir directory of a bucket named examplebucket contains two objects named a.txt and b.txt and a subdirectory named C. A local file named d.txt is stored in the destdir directory of the local root directory. The following structure shows the files and directories in your computer and OSS before synchronization:

examplebucket           Local root directory
└── localdir/             └── destdir/
       ├── a.txt                └── d.txt
       ├── b.txt
       └── C/
  • Synchronize the localdir directory from examplebucket to your computer.

    ./ossutil64 sync  oss://examplebucket/localdir/  destdir/ 

    After you run the preceding command, the a.txt and b.txt objects and the C directory are added to the destfolder directory of your computer. The following structure shows the files and directories in your computer and OSS after synchronization:

    examplebucket           Local root directory
    └── localdir/             └── destdir/
           ├── a.txt                ├── a.txt 
           ├── b.txt                ├── b.txt
           └── C/                   ├── d.txt
                                       └── C/
  • Synchronize the localdir directory of the examplebucket bucket to a local directory named destdir. The --backup-dir option is set to a directory named backup in the command to specify that all objects exist in the destdir directory but do not exist in the localdir directory are deleted from the destdir directory and saved in the backup directory.

    ./ossutil64 sync oss://examplebucket/localdir/  destdir/  --delete  --backup-dir backup/

    After you run the command, the localdir directory of the examplebucket bucket is synchronized to the destdir directory in your computer. Objects that exist in the destdir directory and do not exist in the localdir directory are moved to the backup directory. Only the synchronized objects named a.txt and b.txt and the subdirectory named C are stored in the destdir directory after synchronization. The d.txt object that exists in the destdir directory before synchronization is moved to the backup directory. The following structure shows the files and directories in your computer and OSS after synchronization by specifying the backup directory:

    examplebucket              Local root directory
    └── localdir/               ├── destdir/
           ├── a.txt            │     ├── a.txt 
           ├── b.txt            │     ├── b.txt
           └── C/               │     └── C/                             
                                   └── backup/
                                          └── d.txt
  • If the preceding commands run successfully, an output similar to the following one is returned to indicate the number of synchronized objects, the sizes of synchronized objects, and the time consumed by the synchronization task:

    Succeed: Total num: 2, size: 750,081. OK num: 2(upload 2 files).
    
    average speed 1641000(byte/s)

Common options

If you use ossutil to switch to a bucket that is located in another region, add the -e option to the command to specify the endpoint of the region in which the specified bucket is located. If you use ossutil to switch to a bucket that belongs to another Alibaba Cloud account, you can add the -i option to the command to specify the AccessKey ID of the specified account, and add the -k option to the command to specify the AccessKey secret of the specified account.

For example, you can run the following command to synchronize a directory named srcfolder from a bucket named examplebucket that is located in the China (Shanghai) region and owned by another Alibaba Cloud account, to a local directory named examplefolder:

./ossutil64 sync oss://examplebucket/srcfolder/  examplefolder/ -e oss-cn-shanghai.aliyuncs.com -i LTAI4Fw2NbDUCV8zYUzA****  -k 67DLVBkH7EamOjy2W5RVAHUY9H****

For more information about other common options that you can use for the sync command, see View options.