You can run the sync command to synchronize objects between Object Storage Service (OSS) buckets in the same region or between directories of the same bucket.

Usage notes

  • Binary name

    Sample command lines in this topic are based on the 64-bit Linux system. For other systems, replace ./ossutil64 in the commands with the corresponding binary name. For more information, see ossutil.

  • Number of synchronized files

    You can run the sync command to synchronize up to one million local files to OSS at a time. If the sync command is used to synchronize more than one million local files to OSS at a time, the over max sync numbers 1000000. error message is returned.

  • Differences between the sync and cp commands
    • The sync command synchronizes objects in a specified directory by recursively synchronizing all objects and subdirectories in the directory. The cp command recursively synchronizes objects only when the -r option is specified.
    • If you run the sync command to synchronize data to an OSS bucket, you can add the --delete option to the command to delete all objects that do not exist in the source from the destination. In this case, only the synchronized objects are retained in the destination. The cp command does not support the --delete option.
    • The sync command does not support the --version-id option. Therefore, you cannot run the sync command to synchronize previous versions in versioned buckets. The cp command supports the --version-id option.

    Except for the preceding differences, the sync command can be run in the same way as the cp command. For more information about how to run the cp command, see Copy objects.

Command syntax

./ossutil64 sync cloud_url cloud_url
[-f --force]
[-u --update]
[--delete]
[--enable-symlink-dir]
[--disable-all-symlink]
[--disable-ignore-error]
[--only-current-dir]
[--output-dir <value>]
[--bigfile-threshold <value>]
[--part-size <value>]
[--checkpoint-dir <value>]
[--encoding-type <value>]
[--include <value>]
[--exclude <value>]
[--meta <value>]
[--acl <value>]
[--maxupspeed <value>]
[--disable-crc64]
[--payer <value>]
[-j, --job <value>]
[--parallel <value>]
[--retry-times <value>]
[--tagging <value>]

The following table describes the parameters that you can configure to run the sync command to synchronize objects between OSS paths.

Parameter Description
cloud_url Specifies the paths of the source directory and destination directory. The paths are in the following format: oss://bucketname/path. For example, to synchronize the srcdir directory of a bucket named examplebucket to the destdir directory of the same bucket, set the source directory to oss://examplebucket/srcdir/ and the destination directory to oss://examplebucket/destcdir/. If the value of cloud_url does not end with a forward slash (/), ossutil automatically adds one to the end of the value.
-f --force Forces an operation without prompting the user for confirmation.
-u, --update Specifies that ossutil synchronizes objects from the source only when the objects do not exist in the destination or when the last modified time of the object is later than that of the corresponding objects in the destination.
--delete Specifies that only the synchronized objects are retained in the destination path. Other objects in the destination path are deleted.
Warning To add the --delete option to the command, we recommend that you enable versioning for the bucket to prevent data from being accidentally deleted. For more information about versioning, see Overview.
--enable-symlink-dir Specifies whether to synchronize subdirectories to which symbolic links point.
--disable-all-symlink Specifies that all objects and subdirectories to which symbolic links in the directory point are not synchronized.
--disable-ignore-error Specifies that errors are not ignored during batch operations.
--only-current-dir Specifies that only objects in the current directory are synchronized. Subdirectories in the current directory and objects in these subdirectories are not synchronized.
--output-dir Specifies the directory in which output objects are stored. Output objects are reports that are generated when errors occur during batch synchronization of objects. By default, these reports are stored in the ossutil_output directory of the current directory.
-bigfile-threshold Specifies the maximum size of objects that can be synchronized by using resumable upload or download. Unit: byte.

Default value: 104857600 (100 MB).

Valid values: 0 to 9223372036854775807.

--part-size Specifies the part size. Unit: byte. By default, ossutil determines the part size based on the object size.

Valid values: 1 to 9223372036854775807.

--checkpoint-dir Specifies the directory in which the log information of resumable upload or download tasks is stored. Default value: .ossutil_checkpoint. When a resumable synchronization task fails, ossutil creates this directory and stores the checkpoint information about the task in this directory. ossutil deletes the directory after the task is completed. Therefore, if you specify a checkpoint directory, make sure that ossutil can delete the directory.
--encoding-type Specifies the method used to encode the names of objects. Set the value to url. If you do not specify this parameter, the names of objects are not encoded.
--include Specifies that the command applies to all objects that meet the specified conditions.
--exclude Specifies that the command applies to all objects that do not meet the specified conditions.
--meta Specifies the metadata of an object in the header:value#header:value format. Example: Cache-Control:no-cache#Content-Encoding:gzip. For more information about object metadata, see set-meta.
--acl Specifies the access control list (ACL) of an object. Default value: private. Valid values:
  • default: The ACL of the objects is the same as the ACL of the bucket in which the objects are stored.
  • private: Only the bucket owner can perform read and write operations on objects in the bucket. Other users cannot access the objects in the bucket.
  • public-read: Only the bucket owner can perform write operations on objects in the bucket. Other users, including anonymous users, can perform only read operations on the objects in the bucket. If you set the object ACL to this value, data leakage may occur and you may be charged additional fees. If a user writes illegal information to your objects, your legitimate interests and rights may be infringed. Therefore, we recommend that you do not set the object ACL to this value except in special cases.
  • public-read-write: All users, including anonymous users, can perform read and write operations on the objects in the bucket. If you set the object ACL to this value, data leakage may occur and you may be charged additional fees. Exercise caution when you set the bucket ACL to this value.
--maxupspeed Specifies the maximum upload speed. Unit: KB/s. Default value: 0. The value of 0 indicates that the upload speed is not limited.
--disable-crc64 Disables CRC-64 in synchronization.
--payer Specifies the payer of the traffic and request fees charged when the command is run. If you want the requester who accesses the resources in the specified path to pay for the traffic and request fees charged during queries, set this parameter to requester.
-j, --job Specifies the number of concurrent tasks performed across multiple objects. Valid values: 1 to 10000. Default value: 3.
--parallel Specifies the number of concurrent tasks performed for a single object. Valid values: 1 to 10000. By default, if you do not set this parameter, ossutil sets the value of this parameter based on the operation type and object size.
--retry-times Specifies the number of retries after the command fails to be run. Default value: 10. Valid values: 1 to 500.
--tagging Specifies the tags of objects in the following format: TagkeyA=TagvalueA&TagkeyB=TagvalueB.....

Examples

Your Alibaba Cloud account owns two buckets named examplebucket1 and examplebucket2. The examplebucket1 bucket contains two directories named example and srcdir. The examplebucket bucket contains a directory named destdir. The following structure shows the objects and directories in the two buckets before synchronization:
examplebucket1          examplebucket2
├── exampledir/            └── destdir/
│      ├── a.txt               ├── c.txt
│      └── b.txt               └── e.txt
└── srcdir/     
        └── d.txt       

The following examples show how to run the sync command to synchronize objects between OSS paths in different scenarios:

  • Synchronize the exampledir directory of the examplebucket1 bucket to the srcdir directory of the same bucket.
    ./ossutil64 sync oss://examplebucket1/exampledir/  oss://examplebucket1/srcdir/
    After you run the preceding command, all objects in the exampledir directory are synchronized to the srcdir directory. The following structure shows the objects and directories in examplebucket1 after synchronization:
    examplebucket1                    examplebucket1         
    ├── exampledir/                  ├── exampledir/
    │      ├── a.txt                │      ├── a.txt 
    │      └── b.txt                │      └── b.txt 
    └── srcdir/                      └── srcdir/    
            └── d.txt                         ├── a.txt
                                                 ├── b.txt
                                                 └── d.txt     
  • Synchronize the exampledir directory of the examplebucket1 bucket to the destdir directory of the examplebucket2 bucket.
    ./ossutil64 sync oss://examplebucket1/exampledir/ oss://examplebucket2/destdir/
    After you run the preceding command, all objects in the exampledir directory of examplebucket1 are synchronized to the destdir directory of examplebucket2. The following structure shows the objects and directories in examplebucket1 and examplebucket2 after synchronization:
    examplebucket1          examplebucket2
    ├── exampledir/            └── destdir/
    │      ├── a.txt               ├── a.txt  
    │      └── b.txt               ├── b.txt
    └── srcdir/                     ├── c.txt 
            └── d.txt               └── e.txt
  • Specify the --delete parameter to synchronize all objects from examplebucket1 to examplebucket2 and delete objects exist in examplebucket2 but do not exist in examplebucket1.
    ./ossutil64 sync oss://examplebucket1 oss://examplebucket2 --delete
    After you run the preceding command, the exampledir and test directories of examplebucket1 are synchronized to examplebucket2. The c.txt and e.txt objects that exist in examplebucket2 before synchronization are deleted. The following structure shows the objects and directories in examplebucket1 and examplebucket2 after synchronization:
    examplebucket1          examplebucket2
    ├── exampledir/            ├── exampledir/
    │      ├── a.txt          │      ├── a.txt  
    │      └── b.txt          │      └── b.txt
    └── srcdir/                └── srcdir/  
            └── d.txt                  └── d.txt                  
  • If the preceding commands are successful, a similar output is returned to indicate the number of synchronized objects, the sizes of synchronized objects, and the time used for the synchronization tasks:
    Succeed: Total num: 2, size: 750,081. OK num: 2(upload 2 files).
    
    average speed 1641000(byte/s)

Common options

To use ossutil to manage buckets that are located in different regions, you can use the -e option to use the endpoint of the specified bucket. To use ossutil to manage buckets that are owned by different Alibaba Cloud accounts, you can use the -i option to use the AccessKey ID of the specified account, and use the -k option to use the AccessKey secret of the specified account.

For example, you can run the following command to synchronize a directory named srcfolder from a bucket named examplebucket, which is located in the China (Shanghai) region and owned by another Alibaba Cloud account, to a directory named examplefolder in a bucket named testbucket:
./ossutil64 sync oss://examplebucket/srcfolder/  oss://testbucket/examplefolder/ -e oss-cn-shanghai.aliyuncs.com -i LTAI4Fw2NbDUCV8zYUzA****  -k 67DLVBkH7EamOjy2W5RVAHUY9H****

For more information about other common options that you can use for the sync command, see Common options.