You can run the sync command to synchronize multiple objects at the same time between Object Storage Service (OSS) buckets that reside in the same region or between directories that belong to the same bucket.
Usage notes
Binary name
This topic provides sample command lines that are based on the 64-bit Linux system. For other systems, replace ./ossutil64 in the commands with the specific binary name. For more information, see ossutil.
Number of local files to synchronize
When you run the sync command without including the --delete option, the number of local files that you can synchronize at a time is unlimited. If you include the --delete option, you can synchronize up to 1 million local files at a time. If the number of local files that you want to synchronize exceeds 1 million, the over max sync numbers 1000000. error message is returned.
Differences between the sync and cp commands
The sync command recursively synchronizes all objects and subdirectories in a specific directory. The cp command recursively synchronizes objects only if the -r option is specified.
If you run the sync command to synchronize data to an OSS bucket, you can add the --delete option to the command to delete all objects that exist in the source bucket but do not exist in the destination bucket. This way, only the synchronized objects are retained in the destination bucket. The cp command does not support the --delete option.
The sync command does not support the --version-id option. Therefore, the sync command cannot be used to synchronize previous versions of objects in versioned buckets. The cp command supports the --version-id option.
Except for the preceding differences, you can use the sync and cp commands in a similar manner. For more information about how to run the cp command, see Copy objects.
Limits
You cannot synchronize Cold Archive or Deep Cold Archive objects from a source bucket to a destination bucket.
Command syntax
./ossutil64 sync cloud_url cloud_url
[-f --force]
[-u --update]
[--delete]
[--enable-symlink-dir]
[--disable-all-symlink]
[--disable-ignore-error]
[--only-current-dir]
[--output-dir <value>]
[--bigfile-threshold <value>]
[--part-size <value>]
[--checkpoint-dir <value>]
[--encoding-type <value>]
[--include <value>]
[--exclude <value>]
[--meta <value>]
[--acl <value>]
[--maxupspeed <value>]
[--disable-crc64]
[--payer <value>]
[-j, --job <value>]
[--parallel <value>]
[--retry-times <value>]
[--tagging <value>]
The following table describes the options that you can configure to synchronize objects by running the sync command.
Option | Description |
cloud_url | The source path and destination path. The paths are in the Important If the value of |
-f --force | Specifies that the operation is forcibly performed. The operation is performed without a prompt for confirmation. |
-u, --update | Specifies that ossutil synchronizes objects from the source bucket only if the destination objects do not exist or if the last modified time of the source objects is later than that of the destination objects. |
--delete | Specifies that only the synchronized objects are retained in the specific path of the destination bucket. Other objects in the path are deleted. Warning To add the --delete option to the command, we recommend that you enable versioning for the bucket to prevent data from being accidentally deleted. For more information about versioning, see Overview. |
--enable-symlink-dir | Specifies whether to synchronize subdirectories to which symbolic links point. |
--disable-all-symlink | Specifies that all objects and subdirectories to which symbolic links in the directory point are not synchronized. |
--disable-ignore-error | Specifies that errors are not ignored during batch operations. |
--only-current-dir | Specifies that only objects in the current directory are synchronized. Subdirectories in the current directory and objects in these subdirectories are not synchronized. |
--output-dir | The directory in which output objects are stored. Output objects are report objects that are generated when errors occur during the synchronization of multiple objects. By default, these report objects are stored in the ossutil_output directory of the current directory. |
-bigfile-threshold | The object size threshold for using resumable upload. Unit: bytes. Default value: 104857600 (100 MB). Valid values: 0 to 9223372036854775807. |
--part-size | The part size. Unit: bytes. By default, ossutil determines the part size based on the object size. Valid values: 1 to 9223372036854775807. |
--checkpoint-dir | The directory in which the checkpoint information about resumable upload tasks is stored. If a resumable upload task fails, ossutil automatically creates a directory named |
--encoding-type | The method that you want to use to encode the names of objects. Set the value to url. If you do not specify this option, the names of objects are not encoded. |
--include | Specifies that the command takes effect on all objects that meet the specified conditions. For more information, see Synchronize objects between OSS paths. |
--exclude | Specifies that the command takes effect on all objects that do not meet the specified conditions. For more information, see Synchronize objects between OSS paths. |
--meta | The object metadata in the |
--acl | The access control list (ACL) of the objects. Valid values:
|
--maxupspeed | The maximum upload speed. Unit: KB/s. Default value: 0. The value 0 specifies that the upload speed is unlimited. |
--disable-crc64 | Specifies that CRC-64 is disabled. |
--payer | The payer of the request. If you want the requester who accesses the resources in the specified path to pay for the traffic and request fees, set this option to requester. |
-j, --job | The number of concurrent tasks performed across multiple objects. Valid values: 1 to 10000. Default value: 3. |
--parallel | The number of concurrent tasks performed on a single object. Valid values: 1 to 10000. If you do not specify this option, ossutil specifies the value of this option based on the operation type and the object size. |
--retry-times | The number of retries after the command fails to be run. Valid values: 1 to 500. Default value: 10. |
--tagging | The tags of objects in the |
Examples
Your Alibaba Cloud account owns two buckets named examplebucket1 and examplebucket2. examplebucket1 contains two directories named exampledir and srcdir. examplebucket2 contains a directory named destdir. The following structure shows the objects and directories in examplebucket1 and examplebucket2 before synchronization:
examplebucket1 examplebucket2
├── exampledir/ └── destdir/
│ ├── a.txt ├── c.txt
│ └── b.txt └── e.txt
└── srcdir/
└── d.txt
The following examples show how to run the sync command to synchronize objects between OSS paths in different scenarios.
Synchronize the exampledir directory of examplebucket1 to the srcdir directory of the same bucket.
./ossutil64 sync oss://examplebucket1/exampledir/ oss://examplebucket1/srcdir/
After you run the preceding command, all objects in the exampledir directory are synchronized to the srcdir directory. The following structure shows the objects and directories in examplebucket1 after synchronization:
examplebucket1 examplebucket1 ├── exampledir/ ├── exampledir/ │ ├── a.txt │ ├── a.txt │ └── b.txt │ └── b.txt └── srcdir/ └── srcdir/ └── d.txt ├── a.txt ├── b.txt └── d.txt
Synchronize the exampledir directory of examplebucket1 to the destdir directory of examplebucket2.
./ossutil64 sync oss://examplebucket1/exampledir/ oss://examplebucket2/destdir/
After you run the preceding command, all objects in the exampledir directory of examplebucket1 are synchronized to the destdir directory of examplebucket2. The following structure shows the objects and directories in examplebucket1 and examplebucket2 after synchronization:
examplebucket1 examplebucket2 ├── exampledir/ └── destdir/ │ ├── a.txt ├── a.txt │ └── b.txt ├── b.txt └── srcdir/ ├── c.txt └── d.txt └── e.txt
Synchronize all objects from examplebucket1 to examplebucket2 and use the --delete option to delete the objects that exist in examplebucket2 but do not exist in examplebucket1.
./ossutil64 sync oss://examplebucket1 oss://examplebucket2 --delete
After you run the preceding command, the exampledir directory of examplebucket1 is synchronized to examplebucket2. The c.txt and e.txt objects that exist in examplebucket2 before synchronization are deleted. The following structure shows the objects and directories in examplebucket1 and examplebucket2 after synchronization:
examplebucket1 examplebucket2 ├── exampledir/ ├── exampledir/ │ ├── a.txt │ ├── a.txt │ └── b.txt │ └── b.txt └── srcdir/ └── srcdir/ └── d.txt └── d.txt
If the preceding commands are run successfully, an output similar to the following one is returned to indicate the number of synchronized objects, the sizes of synchronized objects, and the time consumed by the synchronization tasks:
Succeed: Total num: 2, size: 750,081. OK num: 2(upload 2 files). average speed 1641000(byte/s)
Common options
If you use ossutil to switch to a bucket that is located in another region, add the -e option to the command to specify the endpoint of the region in which the specified bucket is located. If you use ossutil to switch to a bucket that belongs to another Alibaba Cloud account, you can add the -i option to the command to specify the AccessKey ID of the specified account, and add the -k option to the command to specify the AccessKey secret of the specified account.
For example, you can run the following command to synchronize a directory named srcfolder from a bucket named examplebucket to a directory named examplefolder in a bucket named testbucket. The examplebucket bucket is located in the China (Shanghai) region and is owned by another Alibaba Cloud account.
./ossutil64 sync oss://examplebucket/srcfolder/ oss://testbucket/examplefolder/ -e oss-cn-shanghai.aliyuncs.com -i LTAI4Fw2NbDUCV8zYUzA**** -k 67DLVBkH7EamOjy2W5RVAHUY9H****
For more information about other common options that you can use for the sync command, see View options.