The sync command can be used to synchronize local files to Object Storage Service (OSS).

Usage notes

  • Binary name

    Sample command lines in this topic are based on the 64-bit Linux system. For other systems, replace ./ossutil64 in the commands with the corresponding binary name. For more information, see ossutil.

  • Number of synchronized files

    You can run the sync command to synchronize up to one million local files to OSS at a time. If the sync command is used to synchronize more than one million local files to OSS at a time, the over max sync numbers 1000000. error message is returned.

  • Differences between the sync and cp commands
    • The sync command synchronizes files in a specified directory by recursively synchronizing all files and subdirectories in the directory. The cp command recursively synchronizes files only when the -r option is specified.
    • If you run the sync command to synchronize data to an OSS bucket, you can add the --delete option to the command to delete all objects that do not exist in the source from the destination bucket. In this case, only the synchronized objects are retained in the destination bucket. The cp command does not support the --delete option.
    • The sync command does not support the --version-id option. Therefore, the sync command cannot be used to synchronize previous versions in versioned buckets. The cp command supports the --version-id option.

    Except for the preceding differences, the sync command can be used in the same way as the cp command. For more information about how to use the cp command, see cp (Upload objects).

Command syntax

./ossutil64 sync file_url cloud_url
[-f --force]
[-u --update]
[--delete]
[--enable-symlink-dir]
[--disable-all-symlink]
[--disable-ignore-error]
[--only-current-dir]
[--output-dir <value>]
[--bigfile-threshold <value>]
[--part-size <value>]
[--checkpoint-dir <value>]
[--encoding-type <value>]
[--snapshot-path <value>]
[--include <value>]
[--exclude <value>]
[--meta <value>]
[--acl <value>]
[--maxupspeed <value>]
[--disable-crc64]
[--payer <value>]
[-j, --job <value>]
[--parallel <value>]
[--retry-times <value>]
[--tagging <value>]

The following table describes the parameters that you can configure when you run this command.

Parameter Description
file_url The path of the local files that you want to synchronize to OSS. Examples: /localfolder/ in Linux and D:\localfolder\ in Windows.
cloud_url The path to which you want to synchronize the local files. The path is in the following format: oss://bucketname/path. Example: oss://examplebucket/exampledir/. If the value of cloud_url does not end with a forward slash (/), ossutil automatically adds one to the end of the value.
-f --force Specifies the command to forcibly run without prompting the user for confirmation.
-u, --update Specifies that ossutil synchronizes files from the source only when the files do not exist in the destination or when the last modified time of the files is later than that of the corresponding objects in the destination bucket.
--delete Specifies that only the synchronized objects are retained in the destination bucket. Other objects in the destination bucket are deleted.
Warning To add the --delete option to the command, we recommend that you enable versioning for the destination bucket to prevent data from being accidentally deleted. For more information about versioning, see Overview.
--enable-symlink-dir Specifies whether to synchronize subdirectories to which symbolic links point.
--disable-all-symlink Specifies that all objects and subdirectories to which symbolic links in the directory point are not synchronized.
--disable-ignore-error Specifies that errors are not ignored during batch operations.
--only-current-dir Specifies that only objects in the current directory are synchronized. Subdirectories in the current directory and objects in these subdirectories are not synchronized.
--output-dir Specifies the directory in which output objects are stored. Output objects are reports that are generated when errors occur when you batch synchronize objects. By default, these reports are stored in the ossutil_output directory of the current directory.
-bigfile-threshold Specifies the maximum size of objects that can be synchronized by using resumable upload or download. Unit: byte.

Default value: 104857600 (100 MB).

Valid values: 0 to 9223372036854775807.

--part-size Specifies the part size. Unit: byte. By default, ossutil determines the part size based on the object size.

Valid values: 1 to 9223372036854775807.

--checkpoint-dir Specifies the directory in which checkpoint information is stored. Default value: .ossutil_checkpoint. When a resumable synchronization task fails, ossutil creates this directory and stores the checkpoint information about the task in this directory. ossutil deletes the directory after the task is complete. Therefore, if you specify a checkpoint directory, make sure that ossutil can delete the directory.
--encoding-type Specifies the method used to encode the names of objects. Valid values: url. If you do not specify this parameter, the names of objects are not encoded.
--snapshot-path Specifies the directory in which the snapshots of synchronized objects are stored. In the next synchronization task, ossutil reads the snapshots in this directory for incremental synchronization.
--include Specifies that the command applies to all objects that meet the specified conditions.
--exclude Specifies that the command applies to all objects that do not meet the specified conditions.
--meta Specifies the metadata of an object in the header:value#header:value format. Example: Cache-Control:no-cache#Content-Encoding:gzip. For more information about object metadata, see set-meta.
--acl Specifies the access control list (ACL) of an object. Valid values:
  • default: The ACL of the objects is the same as the ACL of the bucket in which the objects are stored.
  • private: Only the bucket owner can perform read and write operations on objects in the bucket. Other users cannot access the objects in the bucket.
  • public-read: Only the bucket owner can perform write operations on objects in the bucket. Other users, including anonymous users, can perform only read operations on the objects in the bucket. If you set the object ACL to this value, data leakage may occur and you may be charged additional fees. If a user writes illegal information to your objects, your legitimate interests and rights may be infringed. Therefore, we recommend that you do not set the object ACL to this value except in special cases.
  • public-read-write: All users, including anonymous users, can perform read and write operations on the objects in the bucket. If you set the object ACL to this value, data leakage may occur and you may be charged additional fees. Exercise caution when you set the bucket ACL to this value.
--maxupspeed Specifies the maximum upload speed. Unit: KB/s. Default value: 0. The value of 0 indicates that the upload speed is not limited.
--disable-crc64 Disables CRC-64 in synchronization.
--payer Specifies the payer who is charged the traffic and request fees when the command is run. If you want the requester who accesses the resources in the specified path to pay for the traffic and request fee incurred during queries, set this parameter to requester.
-j, --job Specifies the number of concurrent tasks performed across multiple objects. Valid values: 1 to 10000. Default value: 3.
--parallel Specifies the number of concurrent tasks performed for a single object. Valid values: 1 to 10000. By default, if you do not set this parameter, ossutil sets the value of this parameter based on the operation type and object size.
--retry-times Specifies the number of retries after the command fails to be run. Default value: 10. Valid value: 1 to 500.
--tagging Specifies the tags of objects in the following format: TagkeyA=TagvalueA&TagkeyB=TagvalueB.....

Examples

Two files named d.txt and e.png are stored in the localfolder directory of the local root directory. The destfolder directory of a bucket named examplebucket contains two objects named a.txt and b.txt and a subdirectory named C. The following structure shows the files and directories in local disks and OSS before synchronization:

Local root directory                  examplebucket
    └── localfolder         └── destfolder/
            ├── d.txt               ├── a.txt
            ├── e.png               ├── b.txt
                                       └── C/

The following examples show how to use the sync command to synchronize local files to OSS in different scenarios:

  • Synchronize the localfolder directory from the local disk to examplebucket.
    ./ossutil64 sync localfolder/  oss://examplebucket/destfolder/
    After you run the preceding command, the d.txt and e.png objects are added to the destfolder directory of examplebucket. The following structure shows the files and directories in local disks and OSS after synchronization:
    Local root directory                  examplebucket
        └── localfolder         └── destfolder/
                ├── d.txt               ├── a.txt
                ├── e.png               ├── b.txt
                                           ├── d.txt 
                                           ├── e.png 
                                           └── C/
                            
  • Synchronize the localfolder directory from the local disk to examplebucket and retain only the synchronized objects in the destfolder directory of examplebucket.

    You can add the --delete option in the command to delete all objects that do not exist in localfolder from the destfolder. In this case, only the synchronized objects are retained in destfolder.

    ./ossutil64 sync localfolder/  oss://examplebucket/destfolder/ --delete

    After you run the command, the localfolder directory is synchronized from the local disk to examplebucket. The a.txt and b.txt objects and the C subdirectory within the destfolder directory of examplebucket are deleted. Only the synchronized d.txt and e.png files are retained in the destfolder directory of examplebucket. The following structure shows the files and directories in local disks and OSS after synchronization:

    Local root directory                  examplebucket
        └── localfolder         └── destfolder/
                ├── d.txt                ├── d.txt
                ├── e.png                ├── e.png 
  • Synchronize the localfolder from the local disk to examplebucket without confirmation.

    By default, when you synchronize a local folder to an OSS bucket, if the OSS bucket contains objects that have the same name as files in the local directory, OSS prompts you to confirm whether you want to overwrite the existing objects. To receive confirmation during synchronization, you can run the following command to synchronize the localfolder directory from the local disk to examplebucket:

    ./ossutil64 sync localfolder/ oss://examplebucket/destfolder/
    cp: overwrite "oss://examplebucket/destfolder/d.txt"(y or N)?

    If you confirm that the objects in examplebucket can be overwritten, you can add the -f,--force option to the command to force the overwrite operation. You can run the following command to synchronize the localfolder directory from the local disk to examplebucket without confirmation:

    ./ossutil64 sync localfolder/ oss://examplebucket/destfolder/ -f,--force

    After you run the command, the d.txt and e.png objects are added to the destfolder directory of examplebucket. The following structure shows the files and directories in local disks and OSS after synchronization:

    Local root directory                  examplebucket
        └── localfolder         └── destfolder/
                ├── d.txt               ├── a.txt
                ├── e.png               ├── b.txt
                                           ├── d.txt 
                                           ├── e.png
                                           └── C/
                        
  • If the preceding commands are successful, an output similar to the following is returned to indicate the number of synchronized objects, the sizes of synchronized objects, and the time used for the synchronization tasks:
    Succeed: Total num: 2, size: 750,081. OK num: 2(upload 2 files).
    
    average speed 1641000(byte/s)

Common options

To use ossutil to manage buckets that are located in different regions, you can use the -e option to use the endpoint of the specified bucket. To use ossutil to manage buckets that are owned by different Alibaba Cloud accounts, you can use the -i option to use the AccessKey ID of the specified account, and use the -k option to use the AccessKey secret of the specified account.

For example, you can run the following command to synchronize local files in a local directory named srcfolder to the testfolder directory of a bucket named examplebucket, which is located in the China (Shanghai) region and owned by another Alibaba Cloud account.
./ossutil64 sync srcfolder/ oss://examplebucket/testfolder/ -e oss-cn-shanghai.aliyuncs.com -i LTAI4Fw2NbDUCV8zYUzA****  -k 67DLVBkH7EamOjy2W5RVAHUY9H****

For more information about other common options, see Common options.