How to synchronize local files to OSS - Object Storage Service

Use the sync command to synchronize local files or folders to OSS. The sync command compares the last modified time of files or uses snapshot information to incrementally identify and upload only the changed files. This process efficiently keeps the content at the source and destination consistent.

How it works

Number of files
When you run the sync command without the --delete option, there is no limit on the number of files that you can synchronize at a time. If you include the --delete option, you can synchronize up to 1 million files at a time. If the number of files to synchronize exceeds 1 million, the over max sync numbers 1000000. error is returned.
Differences between the sync and cp commands
- The sync command recursively traverses all files and subdirectories in a specified folder. The cp command performs a recursive operation only if you add the -r option.
- When you use the sync command to synchronize data to OSS, you can add the --delete option to delete files that exist at the destination but not at the source. This option ensures that the destination directory contains only the files from the current synchronization. The cp command does not support the --delete option.
- The sync command does not support the --version-id option. You cannot use the sync command to synchronize previous versions of files in a bucket that has versioning enabled. The cp command supports the --version-id option.
Other than these differences, the sync and cp commands are used in a similar way. For more information about the usage and examples of the cp command, see cp (upload files).

Usage note

Starting with ossutil version 1.6.16, you can use the ossutil binary directly without renaming it for your operating system. For versions earlier than 1.6.16, you must rename the binary file. For more information, see ossutil Command Reference.

Permissions

By default, only an Alibaba Cloud account has the permissions to perform all API operations. To execute this command, a RAM user or RAM role must be granted the required permissions by an Alibaba Cloud account or an administrator through a RAM Policy or a Bucket Policy.

API Action	Description
oss:ListObjects	Lists objects in the destination bucket to compare with local files.
oss:PutObject	Uploads files to OSS.
oss:DeleteObject	[Optional] Required when you use the --delete option to delete extra files from the destination bucket.
oss:PutObjectTagging	[Optional] Required when you use the --tagging option to add tags to an object.

Command syntax

ossutil sync file_url cloud_url [options]

The following table describes the parameters and options.

Parameter	Description
file_url	The path of the on-premises folder to synchronize. For example, `/localfolder/` for Linux or `D:\localfolder\` for Windows.
cloud_url	The path of the destination OSS folder. The format is `oss://bucketname/path/`. For example, `oss://examplebucket/exampledir/`. If the entered `cloud_url` does not end with a forward slash (/), ossutil automatically adds one to the end.
-f --force	Forces the operation without a confirmation prompt.
-u, --update	Synchronizes a file only if the destination file does not exist, or if the last modified time of the source file is later than that of the destination file.
--delete	Deletes other files in the destination path and retains only the files from the current synchronization. Warning Before you use the --delete option, enable versioning to prevent accidental data deletion.
--enable-symlink-dir	Synchronizes linked subdirectories.
--disable-all-symlink	Ignores all linked files and linked subdirectories during folder synchronization.
--disable-ignore-error	Does not ignore errors during batch operations.
--only-current-dir	Synchronizes only the files in the current directory. Subdirectories and the files within them are ignored.
--output-dir	Specifies the directory to store output files. Output files are report files generated when errors occur during batch synchronization. By default, these reports are saved to the ossutil_output directory in the current directory.
-bigfile-threshold	The file size threshold for resumable upload. Unit: bytes. Default value: 100 MB Valid values: 0 to 9223372036854775807
--part-size	The part size. Unit: bytes. By default, ossutil calculates a suitable part size based on the file size. Valid values: 1 to 9223372036854775807
--checkpoint-dir	Specifies the directory to store checkpoint information for resumable uploads. When a resumable upload fails, ossutil automatically creates a directory named `.ossutil_checkpoint` to record checkpoint information. This directory is deleted after the upload succeeds. If you specify a directory for this option, ensure that the specified directory can be deleted.
--encoding-type	The encoding format of file names. Set the value to url. If you do not specify this option, file names are not encoded.
--snapshot-path	Specifies the directory to save snapshot information for the synchronization. In the next synchronization task, ossutil reads the snapshot information from this directory to perform an incremental synchronization.
--include	Includes all files that meet the specified condition. For more information, see include and exclude options.
--exclude	Excludes all files that meet the specified condition. For more information, see include and --exclude.
--meta	Sets the metadata of files. The format is `header:value#header:value`. For example: `Cache-Control:no-cache#Content-Encoding:gzip`. For more information, see set-meta (Manage file metadata).
--acl	The access control list (ACL) of the files. Valid values: default: The ACL of the files is inherited from the bucket. private (default): Only the bucket owner has read and write permissions on the files. Other users cannot access the files. public-read: Only the bucket owner has write permissions on the files. Other users, including anonymous users, have only read permissions on the files. This may expose your data and increase your fees. If malicious users write illegal information to your bucket, your legal rights and interests may be infringed. We recommend that you do not configure this permission except for special scenarios. public-read-write: All users, including anonymous users, have read and write permissions on the files. This may expose your data and increase your fees. Use this permission with caution.
--maxupspeed	The maximum upload speed. Unit: KB/s. The default value is 0, which indicates that the upload speed is not limited.
--disable-crc64	Disables CRC-64 data validation.
--payer	The payment method for the request. If you want the requester to pay for fees such as traffic and requests generated by accessing the resources in the specified path, set this option to requester.
-j，--job	The number of concurrent tasks for multi-file operations. Default value: 3. Valid values: 1 to 10000.
--parallel	The number of concurrent tasks for a single-file operation. Valid values: 1 to 10000. If you do not specify this option, ossutil determines the value based on the operation type and file size.
--retry-times	The number of retries when an error occurs. Default value: 10. Valid values: 1 to 500.
--tagging	The tag information of the files. The format is `TagkeyA=TagvalueA&TagkeyB=TagvalueB....`.

For more information about other common options for this command, see Common options.

Examples

The following examples use a Linux system. You must modify the parameters to match your operating system and environment. The examples assume the following environment:

Two files, d.txt and e.png, are in the local localfolder directory. The destfolder directory in a bucket named examplebucket contains two files, a.txt and b.txt, and a subdirectory named C. The initial content is as follows.

Local root directory                  examplebucket
    └── localfolder         └── destfolder/
            ├── d.txt               ├── a.txt
            └── e.png               ├── b.txt
                                    └── C/

Synchronize a local folder to OSS

Synchronize all files from the local `localfolder/` directory to the destination `destfolder/` directory. This operation uploads only new files. Existing files, such as `a.txt` and `b.txt`, and the `C/` subdirectory in the destination bucket are retained.

ossutil sync localfolder/ oss://examplebucket/destfolder/

After the synchronization is successful, the output includes the number of synchronized files, the total file size, and the time taken for the operation:

Succeed: Total num: 2, size: 750,081. OK num: 2(upload 2 files).

average speed 1641000(byte/s)

After synchronization, the structure of the `destfolder/` directory is as follows:

destfolder/
├── a.txt
├── b.txt
├── d.txt 
├── e.png 
└── C/

Delete extra files in the destination directory during synchronization

When you synchronize the local `localfolder/` directory, you can use the --delete option to delete files and directories, such as `a.txt`, `b.txt`, and `C/`, from the `destfolder/` directory that are not in the source.

ossutil sync localfolder/ oss://examplebucket/destfolder/ --delete

After synchronization, the directory structure of `destfolder/` is identical to `localfolder/`:

destfolder/
├── d.txt
└── e.png

Filter and synchronize specific files

You can use the --include and --exclude options to filter the files to synchronize.

For example, the following command synchronizes only .txt files.
```
ossutil sync localfolder/  oss://examplebucket/destfolder/ --include "*.txt"
```
After you run the command, only d.txt is uploaded, and e.png is ignored. Existing files in the destination bucket are not affected.
```
destfolder/
├── a.txt
├── b.txt
├── d.txt 
└── C/
```
The following command synchronizes only non-.txt files.
```
ossutil sync localfolder/  oss://examplebucket/destfolder/ --exclude "*.txt"
```
After you run the command, only e.png is uploaded, and d.txt is ignored. Existing files in the destination bucket are not affected.
```
destfolder/
├── a.txt
├── b.txt
├── e.png 
└── C/
```

Forcibly overwrite files with the same name in the destination directory

By default, if a file with the same name as a file that you are uploading exists in the destination directory, ossutil prompts you to confirm the overwrite. If you are sure that you want to overwrite all destination files with the same names, you can use the -f or --force option to skip the confirmation prompt and forcibly overwrite the files.

ossutil sync localfolder/ oss://examplebucket/destfolder/ -f

After the command is run successfully, the output includes information such as the number of synchronized files, the total size, and the average speed. The following is a sample output:

Succeed: Total num: 2, size: 750,081. OK num: 2(upload 2 files).

average speed 1641000(byte/s)

Cross-account synchronization

Use the -e, -i, and -k options to synchronize files from the local `srcfolder` directory to the `testfolder` directory in the `examplebucket` bucket. This bucket is in the China (Shanghai) region and is owned by another Alibaba Cloud account.

Note

Specify the endpoint for the region of your bucket. For more information, see Regions and Endpoints.

ossutil sync srcfolder/ oss://examplebucket/testfolder/ -e oss-cn-shanghai.aliyuncs.com -i LTAI4Fw2NbDUCV8zYUzA**** -k 67DLVBkH7EamOjy2W5RVAHUY9H****

Security tip: Including an AccessKey in a command poses a security risk. For automated or long-term tasks, we recommend that you create a RAM role for the source account and grant permissions to the destination account. This method provides more secure cross-account access.