You can run the sync command to synchronize multiple objects at the same time between Object Storage Service (OSS) buckets that reside in the same region or between directories that belong to the same bucket.

Usage notes

  • Binary name

    This topic provides sample command lines that are based on the 64-bit Linux system. For other systems, replace ./ossutil64 in the commands with the corresponding binary name. For more information, see ossutil.

  • Number of synchronized files

    When you run the sync command without including the --delete option, the number of files that you can synchronize at a time is unlimited. If you include the --delete option, you can synchronize up to 1 million files at a time. If the number of files to synchronize exceeds 1 million, the over max sync numbers 1000000. error message is returned.

  • Differences between the sync and cp commands
    • The sync command recursively synchronizes all objects and subdirectories in a specified directory. The cp command recursively synchronizes objects only when the -r option is specified.
    • If you run the sync command to synchronize data to an OSS bucket, you can add the --delete option to the command to delete all objects that do not exist in the source bucket but exist in the destination bucket. This way, only the synchronized objects are retained in the destination bucket. The cp command does not support the --delete option.
    • The sync command does not support the --version-id option. You cannot use the sync command to synchronize previous versions of objects in buckets for which versioning is enabled. The cp command supports the --version-id option.

    Except for the preceding differences, you can use the sync and cp commands in a similar manner. For more information about how to run the cp command, see Copy objects.

Command syntax

./ossutil64 sync cloud_url cloud_url
[-f --force]
[-u --update]
[--delete]
[--enable-symlink-dir]
[--disable-all-symlink]
[--disable-ignore-error]
[--only-current-dir]
[--output-dir <value>]
[--bigfile-threshold <value>]
[--part-size <value>]
[--checkpoint-dir <value>]
[--encoding-type <value>]
[--include <value>]
[--exclude <value>]
[--meta <value>]
[--acl <value>]
[--maxupspeed <value>]
[--disable-crc64]
[--payer <value>]
[-j, --job <value>]
[--parallel <value>]
[--retry-times <value>]
[--tagging <value>]

The following table describes the parameters that you can configure to synchronize objects by running the sync command.

Parameter Description
cloud_url Specifies the paths of the source directory and destination directory. The path is in the oss://bucketname/path format. For example, if you want to synchronize the srcdir directory from a bucket named examplebucket to the destdir directory that is located in the same bucket, set the source path to oss://examplebucket/srcdir/ and the destination path to oss://examplebucket/destcdir/.
Notice If the value of cloud_url does not end with a forward slash (/), ossutil automatically adds a forward slash (/) to the end of the value.
-f --force Specifies the operation that is forcibly performed without prompts.
-u, --update Specifies that ossutil synchronizes objects from the source bucket only if the objects do not exist in the destination bucket, or if the last modified time of the source objects is later than the modified time of the corresponding objects in the destination bucket.
--delete Specifies that only the synchronized objects are retained in the specified path of the destination bucket. Other objects in the specified path are deleted.
Warning To add the --delete option to the command, we recommend that you enable versioning for the bucket to prevent data from being accidentally deleted. For more information about versioning, see Overview.
--enable-symlink-dir Specifies whether to synchronize subdirectories to which symbolic links point.
--disable-all-symlink Specifies that all objects and subdirectories to which symbolic links in the directory point are not synchronized.
--disable-ignore-error Specifies that errors are not ignored during batch operations.
--only-current-dir Specifies that only the objects in the current directory are synchronized. Subdirectories in the current directory and the objects in these subdirectories are not synchronized.
--output-dir Specifies the directory in which output objects are stored. Output objects are report objects that are generated when errors occur during the synchronization of multiple objects. By default, these report objects are stored in the ossutil_output directory of the current directory.
-bigfile-threshold Specifies the maximum size of objects that can be uploaded by using resumable upload. Unit: bytes.

Default value: 104857600 (100 MB).

Valid values: 0 to 9223372036854775807.

--part-size Specifies the part size. Unit: bytes. By default, ossutil determines the part size based on the object size.

Valid values: 1 to 9223372036854775807.

--checkpoint-dir Specifies the directory in which the log information of resumable upload operations is stored. If a resumable upload task fails, ossutil automatically creates the .ossutil_checkpoint directory, and stores the checkpoint information in the directory. When the resumable upload task is complete, ossutil deletes the .ossutil_checkpoint directory. If you specify a checkpoint directory, make sure that the directory can be deleted.
--encoding-type Specifies the method that is used to encode the names of objects. Valid value: url. If you do not set this parameter, the names of objects are not encoded.
--include Specifies that the command takes effect on all objects that meet the specified conditions.
--exclude Specifies that the command takes effect on all objects that do not meet the specified conditions.
--meta Specifies the metadata of an object, which is in the header:value#header:value format. Example: Cache-Control:no-cache#Content-Encoding:gzip. For more information about object metadata, see set-meta.
--acl Specifies the access control list (ACL) of an object. Default value: private. Valid values:
  • default: The ACLs of the objects are the same as the ACL of the bucket in which the objects are stored.
  • private: Only the bucket owner can perform read and write operations on the objects in the bucket. Other users cannot access the objects in the bucket.
  • public-read: Only the bucket owner can perform write operations on the objects in the bucket. Other users, including anonymous users, can perform only read operations on the objects in the bucket. This may result in unauthorized access to the data in your bucket and high fees. If a user uploads prohibited data or information, your legal rights may be infringed. We recommend that you do not set the object ACL to public-read except in special cases.
  • public-read-write: All users, including anonymous users, can perform read and write operations on the objects in the bucket. This may result in unauthorized access to the data in your bucket and high fees. Proceed with caution when you set this parameter to public-read-write.
--maxupspeed Specifies the maximum upload speed. Unit: KB/s. Default value: 0. The value 0 specifies that the upload speed is unlimited.
--disable-crc64 Specifies that CRC-64 is disabled.
--payer Specifies the payer of the request. If you want the requester who wants to access the resources in the specified path to pay for the traffic and request fees, set this parameter to requester.
-j, --job Specifies the number of concurrent tasks that are performed across multiple objects. Valid values: 1 to 10000. Default value: 3.
--parallel Specifies the number of concurrent tasks that are performed on a single object. Valid values: 1 to 10000. If you do not set this parameter, ossutil specifies the value of this parameter based on the operation type and object size.
--retry-times Specifies the number of retries after the command fails to run. Valid values: 1 to 500. Default value: 10.
--tagging Specifies the tags of objects, which are in the TagkeyA=TagvalueA&TagkeyB=TagvalueB.... format.

Examples

Your Alibaba Cloud account owns two buckets named examplebucket1 and examplebucket2. examplebucket1 has two directories named exampledir and srcdir. examplebucket2 has a directory named destdir. The following structure shows the objects and directories in examplebucket1 and examplebucket2 before synchronization:
examplebucket1          examplebucket2
├── exampledir/            └── destdir/
│      ├── a.txt               ├── c.txt
│      └── b.txt               └── e.txt
└── srcdir/     
        └── d.txt       

The following examples show how to run the sync command to synchronize objects between OSS paths in different scenarios:

  • Synchronize the exampledir directory of examplebucket1 to the srcdir directory that is located in the same bucket.
    ./ossutil64 sync oss://examplebucket1/exampledir/  oss://examplebucket1/srcdir/
    After you run the preceding command, all objects in the exampledir directory are synchronized to the srcdir directory. The following structure shows the objects and directories in examplebucket1 after synchronization:
    examplebucket1                    examplebucket1         
    ├── exampledir/                  ├── exampledir/
    │      ├── a.txt                │      ├── a.txt 
    │      └── b.txt                │      └── b.txt 
    └── srcdir/                      └── srcdir/    
            └── d.txt                         ├── a.txt
                                                 ├── b.txt
                                                 └── d.txt     
  • Synchronize the exampledir directory of examplebucket1 to the destdir directory of examplebucket2.
    ./ossutil64 sync oss://examplebucket1/exampledir/ oss://examplebucket2/destdir/
    After you run the preceding command, all objects in the exampledir directory of examplebucket1 are synchronized to the destdir directory of examplebucket2. The following structure shows the objects and directories in examplebucket1 and examplebucket2 after synchronization:
    examplebucket1          examplebucket2
    ├── exampledir/            └── destdir/
    │      ├── a.txt               ├── a.txt  
    │      └── b.txt               ├── b.txt
    └── srcdir/                     ├── c.txt 
            └── d.txt               └── e.txt
  • Synchronize all objects from examplebucket1 to examplebucket2 and use the --delete option to delete the objects that exist in examplebucket2 but do not exist in examplebucket1.
    ./ossutil64 sync oss://examplebucket1 oss://examplebucket2 --delete
    After you run the preceding command, the exampledir and srcdir directories of examplebucket1 are synchronized to examplebucket2. The c.txt and e.txt objects that exist in examplebucket2 are deleted. The following structure shows the objects and directories in examplebucket1 and examplebucket2 after synchronization:
    examplebucket1          examplebucket2
    ├── exampledir/            ├── exampledir/
    │      ├── a.txt          │      ├── a.txt  
    │      └── b.txt          │      └── b.txt
    └── srcdir/                └── srcdir/  
            └── d.txt                  └── d.txt                  
  • If the preceding commands are successful, the following output is returned to indicate the number of synchronized objects, the sizes of synchronized objects, and the time consumed by the synchronization tasks:
    Succeed: Total num: 2, size: 750,081. OK num: 2(upload 2 files).
    
    average speed 1641000(byte/s)

Common options

If you use ossutil to switch to a bucket that is located in a different region, you can use the -e option to specify the endpoint of the region in which the specified bucket resides. If you use ossutil to switch to a bucket that belongs to another Alibaba Cloud account, you can use the -i option to specify the AccessKey ID of the specified account, and use the -k option to specify the AccessKey secret of the specified account.

For example, you can run the following command to synchronize a directory named srcfolder from a bucket named examplebucket to a directory named examplefolder in a bucket named testbucket. examplebucket is located in the China (Shanghai) region and is owned by another Alibaba Cloud account.
./ossutil64 sync oss://examplebucket/srcfolder/  oss://testbucket/examplefolder/ -e oss-cn-shanghai.aliyuncs.com -i LTAI4Fw2NbDUCV8zYUzA****  -k 67DLVBkH7EamOjy2W5RVAHUY9H****

For more information about other common options that you can use for the sync command, see Common options.