You can run the sync command to synchronize local files to Object Storage Service (OSS).
Usage notes
Binary name
This topic provides sample command lines that are based on the 64-bit Linux system. For other systems, replace ./ossutil64 in the commands with the corresponding binary name. For more information, see ossutil.
Number of local files to synchronize
When you run the sync command without including the --delete option, the number of local files that you can synchronize at a time is unlimited. If you include the --delete option, you can synchronize up to 1 million local files at a time. If the number of local files that you want to synchronize exceeds 1 million, the over max sync numbers 1000000. error message is returned.
Differences between the sync and cp commands
The sync command recursively synchronizes all files and subdirectories in a specific directory. The cp command recursively uploads files only if the -r option is specified.
If you run the sync command to synchronize data from a local source to an OSS bucket, you can add the --delete option to the command to delete all files that do not exist in the local source but exist in the bucket. This way, only the synchronized files are retained in the bucket. The cp command does not support the --delete option.
The sync command does not support the --version-id option. Therefore, the sync command cannot be used to synchronize previous versions of objects in versioning-enabled buckets. The cp command supports the --version-id option.
Except for the preceding differences, you can use the sync and cp commands in a similar manner. For more information about how to run the cp command, see Upload objects.
Command syntax
./ossutil64 sync file_url cloud_url
[-f --force]
[-u --update]
[--delete]
[--enable-symlink-dir]
[--disable-all-symlink]
[--disable-ignore-error]
[--only-current-dir]
[--output-dir <value>]
[--bigfile-threshold <value>]
[--part-size <value>]
[--checkpoint-dir <value>]
[--encoding-type <value>]
[--snapshot-path <value>]
[--include <value>]
[--exclude <value>]
[--meta <value>]
[--acl <value>]
[--maxupspeed <value>]
[--disable-crc64]
[--payer <value>]
[-j, --job <value>]
[--parallel <value>]
[--retry-times <value>]
[--tagging <value>]
The following table describes the options that you can configure to synchronize local files by running the sync command.
Option | Description |
file_url | The path of the local files that you want to synchronize to OSS. Examples: |
cloud_url | The OSS directory to which you want to synchronize the local files. The path is in the |
-f --force | Specifies that the operation is forcibly performed. The operation is performed without a prompt for confirmation. |
-u, --update | Specifies that ossutil synchronizes local files only if the local files do not exist in the bucket or if the last modified time of the local files is later than that of the objects in the bucket. |
--delete | Specifies that only the synchronized objects are retained in the bucket. Other objects in the bucket are deleted. Warning Before you add the --delete option to the command, we recommend that you enable versioning for the bucket to prevent data from being accidentally deleted. For more information about versioning, see Overview. |
--enable-symlink-dir | Specifies that subdirectories to which symbolic links point are synchronized. |
--disable-all-symlink | Specifies that all local files and subdirectories to which symbolic links in the directory point are not synchronized. |
--disable-ignore-error | Specifies that errors are not ignored during batch operations. |
--only-current-dir | Specifies that only local files in the current directory are synchronized. Subdirectories in the current directory and files in these subdirectories are not synchronized. |
--output-dir | The directory in which output objects are stored. Output objects are report objects that are generated when errors occur during the synchronization of multiple objects. By default, these report objects are stored in the ossutil_output directory of the current directory. |
-bigfile-threshold | The file size threshold for using resumable upload. If a file exceeds the value of this parameter, the object is uploaded by using resumable upload. Unit: bytes. Default value: 104857600. This value is equivalent to 100 MB. Valid values: 0 to 9223372036854775807. |
--part-size | The part size. Unit: bytes. By default, ossutil determines the part size based on the file size. Valid values: 1 to 9223372036854775807. |
--checkpoint-dir | The directory in which the checkpoint information about resumable upload tasks is stored. If a resumable upload task fails, ossutil automatically creates a directory named |
--encoding-type | The method that you want to use to encode the names of objects. Set the value to url. If you do not specify this option, the names of objects are not encoded. |
--snapshot-path | The directory in which the snapshots of synchronized objects are stored. In the next synchronization task, ossutil reads the snapshots in this directory for incremental synchronization. |
--include | Specifies that the command takes effect on all local files that meet the specified conditions. For more information, see Synchronize local files to OSS. |
--exclude | Specifies that the command applies to all local files that do not meet the specified conditions. For more information, see Synchronize local files to OSS. |
--meta | The object metadata in the |
--acl | The access control list (ACL) of the objects. Valid values:
|
--maxupspeed | The maximum upload speed. Unit: KB/s. Default value: 0. The value 0 specifies that the upload speed is unlimited. |
--disable-crc64 | Specifies that CRC-64 is disabled. |
--payer | The payer of the request. If you want the requester who accesses the resources in the specified path to pay for the traffic and request fees, set this option to requester. |
-j, --job | The number of concurrent tasks performed across multiple objects. Valid values: 1 to 10000. Default value: 3. |
--parallel | The number of concurrent tasks performed on a single object. Valid values: 1 to 10000. If you do not specify this option, ossutil specifies the value of this option based on the operation type and the object size. |
--retry-times | The number of retries after the command fails to be run. Valid values: 1 to 500. Default value: 10. |
--tagging | The tags of objects in the |
Examples
Two files named d.txt and e.png are stored in the localfolder directory of the local root directory. The destfolder directory of a bucket named examplebucket contains two objects named a.txt and b.txt and a subdirectory named C. The following structure shows the files and directories on your computer and OSS before synchronization:
Local root directory examplebucket
└── localfolder └── destfolder/
├── d.txt ├── a.txt
├── e.png ├── b.txt
└── C/
The following examples show how to run the sync command to synchronize local files to OSS in different scenarios.
Synchronize the localfolder directory from your computer to the examplebucket bucket
./ossutil64 sync localfolder/ oss://examplebucket/destfolder/
After you run the preceding command, the d.txt and e.png objects are added to the destfolder directory of the examplebucket bucket. The following structure shows the files and directories on your computer and OSS after synchronization:
Local root directory examplebucket └── localfolder └── destfolder/ ├── d.txt ├── a.txt ├── e.png ├── b.txt ├── d.txt ├── e.png └── C/
Synchronize the .txt object from the localfolder directory to the examplebucket bucket
./ossutil64 sync localfolder/ oss://examplebucket/destfolder/ --include "*.txt"
After you run the preceding command, the d.txt object is added to the destfolder directory of the examplebucket bucket. The following structure shows the files and directories on your computer and OSS after synchronization:
Local root directory examplebucket └── localfolder └── destfolder/ ├── d.txt ├── a.txt ├── e.png ├── b.txt ├── d.txt └── C/
Synchronize non-TXT objects from the localfolder directory to the examplebucket bucket
./ossutil64 sync localfolder/ oss://examplebucket/destfolder/ --exclude "*.txt"
After you run the preceding command, the e.png object is added to the destfolder directory of the examplebucket bucket. The following structure shows the files and directories on your computer and OSS after synchronization:
Local root directory examplebucket └── localfolder └── destfolder/ ├── d.txt ├── a.txt ├── e.png ├── b.txt ├── e.png └── C/
Synchronize the localfolder directory from your computer to the examplebucket bucket and retain only the synchronized objects in the destfolder directory of the examplebucket bucket
Add the --delete option in the command to delete all objects that do not exist in the localfolder directory from the destfolder directory. This way, only the synchronized objects are retained in the destfolder directory.
./ossutil64 sync localfolder/ oss://examplebucket/destfolder/ --delete
After you run the preceding command, the localfolder directory is synchronized from your computer to the examplebucket bucket. The a.txt and b.txt objects and the C subdirectory in the destfolder directory of the examplebucket bucket are deleted. Only the synchronized d.txt and e.png objects are retained in the destfolder directory of the examplebucket bucket. The following structure shows the files and directories on your computer and OSS after synchronization:
Local root directory examplebucket └── localfolder └── destfolder/ ├── d.txt ├── d.txt ├── e.png ├── e.png
Synchronize the localfolder directory from your computer to the examplebucket bucket without confirmation
By default, when you synchronize a local directory to an OSS bucket, if the OSS bucket contains objects that have the same name as the files in the local directory, OSS prompts you to confirm whether you want to overwrite the existing objects in the bucket, as shown in the following sample command:
./ossutil64 sync localfolder/ oss://examplebucket/destfolder/ cp: overwrite "oss://examplebucket/destfolder/d.txt"(y or N)?
If you confirm that the objects in the examplebucket bucket can be overwritten, you can add the -f,--force option to the command to forcibly overwrite the existing objects in the bucket, as shown in the following sample command:
./ossutil64 sync localfolder/ oss://examplebucket/destfolder/ -f
After you run the preceding command, the d.txt and e.png objects are added to the destfolder directory of the examplebucket bucket. The following structure shows the files and directories on your computer and OSS after synchronization:
Local root directory examplebucket └── localfolder └── destfolder/ ├── d.txt ├── a.txt ├── e.png ├── b.txt ├── d.txt ├── e.png └── C/
If the preceding commands run successfully, an output similar to the following one is returned to indicate the number of synchronized objects, the sizes of synchronized objects, and the time consumed by the synchronization task:
Succeed: Total num: 2, size: 750,081. OK num: 2(upload 2 files). average speed 1641000(byte/s)
Common options
If you use ossutil to switch to a bucket that is located in another region, add the -e option to the command to specify the endpoint of the region in which the specified bucket is located. If you use ossutil to switch to a bucket that belongs to another Alibaba Cloud account, you can add the -i option to the command to specify the AccessKey ID of the specified account, and add the -k option to the command to specify the AccessKey secret of the specified account.
For example, you can run the following command to synchronize files in a local directory named srcfolder to the testfolder directory of a bucket named examplebucket, which is located in the China (Shanghai) region and owned by another Alibaba Cloud account:
./ossutil64 sync srcfolder/ oss://examplebucket/testfolder/ -e oss-cn-shanghai.aliyuncs.com -i LTAI4Fw2NbDUCV8zYUzA**** -k 67DLVBkH7EamOjy2W5RVAHUY9H****
For more information about other common options that you can use for the sync command, see View options.