The sync command is used to synchronize files from the source to the destination.
Usage notes
- The sync command supports the --delete option. When you add this option in the command, if the destination of the synchronization
is an OSS bucket, ossutil deletes all files that do not exist in the source from the
destination bucket and retains only the synchronized files. If the destination of
the synchronization is a local disk, you must also add the --backup-dir option in the command to migrate files that do not exist in the source but do exist
in the destination local disk to the specified folder.
Warning To add the --delete option in the command, we recommend that you enable versioning for the destination bucket to prevent data from being accidentally deleted.
- The sync command synchronizes files in the specified folder by recursively synchronizing all files and subfolders in the folder.
- If the source or destination of the synchronization is an OSS bucket, and the specified folder name does not end with a forward slash, ossutil automatically adds a forward slash (/) to the end of the folder name.
- The sync command does not support the --version-id option and as a result cannot be used to synchronize previous versions in versioned buckets.
Except for the preceding differences, the sync command can be used in the same way as the cp command. For more information about how to use the cp command, see cp.
Synchronize local files to OSS
- Command syntax
The preceding command contains the following parameters:./ossutil sync file_url cloud_url [-f] [-u] [--delete] [--backup-dir] [--enable-symlink-dir] [--disable-all-symlink] [--disable-ignore-error] [--only-current-dir] [--output-dir=odir] [--bigfile-threshold=size] [--checkpoint-dir=cdir] [--snapshot-path=sdir] [--payer requester]
- file_url: The local path in which the files to synchronize are stored. Examples:
/localfolder/
in Linux andD:\localfolder\
in Windows. - cloud_url: The path of the OSS folder to which the local files are synchronized. The value
of this parameter is in the following format:
oss://bucketname/path/
. Example:oss://examplebucket/exampledir/
. If the value ofcloud_url
does not end with a forward slash (/), ossutil automatically adds a forward slash to the end of the value.
- file_url: The local path in which the files to synchronize are stored. Examples:
- Examples
The destination OSS bucket named examplebucket contains a folder named example, which contains two objects named a.txt and b.txt and a subfolder named C. In the root folder of the source local disk, a folder named example contains a file named d.txt. The following figure shows the structures of the destination bucket and the source local disk.
examplebucket root folder of the local disk └── example/ └── example/ ├── a.txt └── d.txt ├── b.txt └── C/
The following examples show how to use the sync command to synchronize local files to OSS in different scenarios:
- Synchronize the example folder in the local disk to the destination OSS bucket.
./ossutil sync example/ oss://examplebucket/example/
After the command is executed, the d.txt file is added to the example folder of the examplebucket bucket.examplebucket root folder of the local disk └── example/ └── example/ ├── a.txt └── d.txt ├── b.txt ├── d.txt └── C/
- Synchronize the example folder in the local disk to the destination OSS bucket without
confirmation.
When you synchronize a local folder to an OSS bucket, if the OSS bucket contains objects that has the same name as files in the local folder, OSS prompts you to confirm whether you want to overwrite the existing objects. If you confirm that the objects can be overwritten, you can add the -f, --force option in the command to force the overwrite operation. You can run the following command to synchronize the example folder in the local disk to the destination OSS bucket without confirmation:
After the command is executed, the d.txt file is added to the example folder of the examplebucket bucket../ossutil sync localfolder/ oss://examplebucket/destfolder/ -f
examplebucket root folder of the local disk └── example/ └── example/ ├── a.txt └── d.txt ├── b.txt ├── d.txt └── C/
- Synchronize the example folder in the local disk to the destination OSS bucket and
delete all objects in that folder of the examplebucket bucket that do not exist within
the same folder in the local disk.
If the destination of the synchronization is an OSS bucket, you can add the --delete option in the command to delete all objects that do not exist in the source from the destination bucket and retains only the synchronized objects.
After the command is executed, the example folder in the local disk is synchronized to the examplebucket bucket. The a.txt and b.txt files and the subfolder C within the example folder of the examplebucket bucket are deleted. Only the synchronized d.txt file is retained in the example folder of the examplebucket bucket../ossutil sync example/ oss://examplebucket/example/ --delete
examplebucket root folder of the local disk └── example/ └── example/ └── d.txt └── d.txt
- Synchronize the example folder in the local disk to the destination OSS bucket.
Synchronize OSS objects to local disks
- Command syntax
You can run the preceding sync command to synchronize objects stored in a bucket to local disks. The command contains the following parameters:./ossutil sync cloud_url file_url [-f] [-u] [--delete] [--backup-dir] [--only-current-dir] [--disable-ignore-error] [--output-dir=odir] [--bigfile-threshold=size] [--checkpoint-dir=cdir] [--range=x-y] [--payer requester]
- file_url: The local path to which OSS objects are synchronized. Examples:
/localfolder/
in Linux andD:\localfolder\
in Windows. - cloud_url: The path of the OSS folder in which the objects to synchronize are stored.
The value of this parameter is in the following format:
oss://bucketname/path/
. Example:oss://examplebucket/exampledir/
.
- file_url: The local path to which OSS objects are synchronized. Examples:
- Examples
The source OSS bucket named examplebucket contains a folder named example, which contains two objects named a.txt and b.txt and a subfolder named C. In the root folder of the destination local disk, a folder named example contains a file named d.txt. The following figure shows the structures of the source bucket and the destination local disk.
examplebucket root folder of the local disk └── example/ └── example/ ├── a.txt └── d.txt ├── b.txt └── C/
The following examples show how to use the sync command to synchronize OSS objects to local disks in different scenarios:
- Synchronize the example folder of the examplebucket bucket to the local folder.
After the command is executed, the a.txt and b.txt files and the subfolder C are added to the example folder in the local disk../ossutil sync oss://examplebucket/example/ example/
examplebucket root folder of the local disk └── example/ └── example/ ├── a.txt ├── a.txt ├── b.txt ├── b.txt └── C/ ├── d.txt └── C/
- Synchronize the example folder of the examplebucket bucket to the local folder, and migrate files that do
not exist in example folder of the examplebucket bucket but do exist in the destination
local folder to the backup folder of the examplebucket bucket.
./ossutil sync oss://examplebucket/example/ example/ --delete --backup-dir backup/
After the command is executed, the example folder of the examplebucket bucket is synchronized to the local disk, and the files in the example folder in the local disk are migrated to the backup folder of the examplebucket bucket. In this example, the a.txt and b.txt files and the subfolder C synchronized from OSS are retained in the example folder in the local disk. The d.txt file is added to the backup folder of the examplebucket bucket.examplebucket root folder of the local disk └── example/ ├── example/ ├── a.txt │ ├── a.txt ├── b.txt │ ├── b.txt └── C/ │ └── C/ └── backup/ └──d.txt
- Synchronize the example folder of the examplebucket bucket to the local folder.
Synchronize objects between OSS buckets
- Command syntax
You can use the preceding sync command to synchronize data between OSS buckets. The source bucket and the destination bucket must be in the same region. The cloud_url parameter in the command indicates the paths of the source bucket and destination bucket. The value of this parameter is in the following format:./ossutil sync cloud_url cloud_url [-f] [-u] [--delete] [--backup-dir] [--only-current-dir] [--disable-ignore-error] [--output-dir=odir] [--bigfile-threshold=size] [--checkpoint-dir=cdir] [--payer requester]
oss://bucketname/path/
. Example:oss://examplebucket/exampledir/
. - Examples
Your Alibaba Cloud account owns two buckets named examplebucket1 and examplebucket2. The examplebucket1 bucket contains the example and test folders. The examplebucket2 bucket contains the example folder. The following figure shows the structures of examplebucket1 and examplebucket2.
examplebucket1 examplebucket2 ├── example/ └── example/ │ ├── a.txt ├── a.txt │ └── b.txt └── c.txt └── test/ └── d.txt
The following examples show how to use the sync command to synchronize data between two OSS buckets in different scenarios:
- Synchronize the example folder of the examplebucket1 bucket to the test folder of the same bucket.
After the command is executed, all objects in the example folder are synchronized to the test folder../ossutil sync oss://examplebucket1/example/ oss://examplebucket1/test/
examplebucket1 examplebucket2 ├── example/ └── example/ │ ├── a.txt ├── a.txt │ └── b.txt └── c.txt └── test/ ├── a.txt ├── b.txt └── d.txt
- Synchronize the example folder of the examplebucket1 bucket to the example folder of the examplebucket2 bucket.
After the command is executed, all objects in the example folder of the exmplebucket1 bucket are synchronized to the example folder of the examplebucket2 bucket../ossutil sync oss://examplebucket1/example/ oss://examplebucket2/example/
examplebucket1 examplebucket2 ├── example/ └── example/ │ ├── a.txt ├── a.txt │ └── b.txt ├── b.txt └── test/ └── c.txt └── d.txt
- Synchronize all objects in examplebucket1 to examplebucket2, and delete objects that
do not exist in examplebucket1 from examplebucket2.
After the command is executed, the example and test folders of the examplebucket1 bucket are synchronized to the root folder of the examplebucket2 bucket. Other objects in the examplebucket2 bucket are deleted../ossutil sync oss://examplebucket1 oss://examplebucket2 --delete
examplebucket1 examplebucket2 ├── example/ ├── example/ │ ├── a.txt │ ├── a.txt │ └── b.txt │ └── b.txt └── test/ └── test/ └── d.txt └── d.txt
- Synchronize the example folder of the examplebucket1 bucket to the test folder of the same bucket.
Common options
Option | Description |
---|---|
-f, --force | Forces an operation without prompting the user for confirmation. |
-u, --update | Specifies that ossutil synchronizes files from the source only when the files do not exist in the destination or when the last modified time of the files is later than that of the corresponding objects in the destination bucket. |
--output-dir | Specifies the directory in which output files are stored.
Output files include the report files generated due to errors that occur when you use the cp and sync commands to copy or synchronize multiple files. Default value: The ossutil_output folder in the current directory. |
--bigfile-threshold | Specifies a threshold of object size. If the size of an object exceeds the threshold,
the object is transmitted by using resumable upload or download. Unit: bytes.
Default value: 104857600 (100 MB). Valid values: 0 to 9223372036854775807. |
--part-size | Specifies the part size. Unit: bytes.
By default, ossutil determines an appropriate part size based on the object size. If you have special requirements or want to optimize the performance, you can set the value of this option. Valid values: 1 to 9223372036854775807. |
--checkpoint-dir | Specifies the checkpoint folder. Default value: .ossutil_checkpoint . If a resumable data transfer task fails, ossutil automatically creates this folder
and records the checkpoint information in the folder. If a resumable data transfer
task succeeds, ossutil deletes the folder. If this option is specified, make sure
that you have the permissions to delete the specified folder.
|
--range | Downloads a specific range of the object as a new object. The minimum start value
of the range is 0, which indicates the byte 0 of the object.
|
--encoding-type | Specifies the encoding type of the names of output or input objects. Valid values:
url.
If this option is not specified in the command, the object name is not encoded.
Note Bucket names cannot be URL-encoded.
|
--include | Specifies that objects whose names contain the specified value are matched. For example,
if the value of this option is *.jpg , objects whose names end with .jpg are considered as matched.
|
--exclude | Specifies that objects whose names do not contain the specified value are matched.
For example, if the value of this option is *.txt , objects whose names do not end with .txt are considered as matched.
|
--meta | Specifies the metadata of an object in the [header:value#header:value...] format. Example: Cache-Control:no-cache#Content-Encoding:gzip . For more information, see set-meta.
|
--acl | Configures the ACL of an object. Valid values:
|
--snapshot-path | If you specify the --snapshot-path option when you synchronize files, ossutil takes a snapshot of the synchronization and stores the snapshot information in a specified folder. When you specify this option in the command to synchronize the files again, ossutil reads the snapshot information from the specified folder and performs incremental synchronization. For more information, see Generate snapshot information. |
--disable-crc64 | Disables CRC64 verification in data transmission.
By default, ossutil performs CRC64 verification in data transmission. |
--maxupspeed | Specifies the maximum upload speed. Unit: Kbit/s.
Default value: 0. The value of 0 indicates that the upload speed is not limited. |
--payer | Specifies the payer of the request. If pay-by-requester is enabled, you can set this
option to requester .
|
-j, --jobs | Specifies the number of operations to perform on multiple objects concurrently.
Default value: 3. Valid values: 1 to 10000. |
--parallel | Specifies the number of operations to perform on a single object concurrently. By
default, ossutil sets the value of this option based on the operation type and object
size.
Valid values: 1 to 10000. |
--loglevel | Specifies the log level. This option is empty by default, which indicates that no
log files are generated. Valid values:
|
--retry-times | Specifies the number that ossutil retires an operation when an error occurs.
Default value: 10 Valid values: 1 to 500. |
--proxy-host | Specifies the URL of the proxy server. HTTP, HTTPS, and SOCKS5 proxies are supported.
Examples: http://120.79. **.**:3128 and socks5://120.79. **. **:1080 .
|
--proxy-user | Specifies the user name of the proxy server. This parameter is empty by default. |
--proxy-pwd | Specifies the password for the proxy server. This parameter is empty by default. |
--local-host | Specifies the local IP address of ossutil. If your computer has multiple IP addresses, you can specify this option so that ossutil can access OSS by using the specified IP address. |
--enable-symlink-dir | Specifies whether to synchronize the subfolders of the symbolic link. By default, the subfolders are not synchronized. The probe command can be used to check whether an object or a folder to which the symbolic link points is also a symbolic link. |
--only-current-dir | Specifies that only objects in the current folder are synchronized. The subfolders in the current folder are not synchronized. |
--disable-dir-object | Specifies that no OSS objects are generated for the folder during synchronization. |
--disable-all-symlink | Specifies that all objects in the subfolder and the subfolder to which the symbolic link points are ignored during object synchronization. |
--tagging | Specifies tags for an object when you synchronize the object. The tags are in the
"abc=1&bcd=2&……" format.
|
--disable-ignore-error | Specifies that errors are not ignored during batch operations. |
--delete | When you add this option in the command, if the destination of the synchronization is an OSS bucket, ossutil deletes all files that do not exist in the source from the destination bucket and retains only the synchronized files. If the destination of the synchronization is a local disk, you must also add the --backup-dir option in the command to migrate files that do not exist in the source but do exist in the destination local disk to the specified folder. |
--backup-dir | This option is specified with the --delete option to back up files that do not exist in the source but exist in the destination local disk to the specified folder when the destination of the synchronization is a local disk. |