Object Storage Service:sync (Synchronize OSS objects to a local computer)

Usage notes

• This operation requires the oss:GetObject and oss:ListObjects permissions. Grant custom permissions to a RAM user.

• Binary name

  For ossutil 1.6.16 and later, you can use ossutil directly as the binary name in the command line. You do not need to update the binary name based on the operating system. If you use a version of ossutil earlier than 1.6.16, update the binary name based on your operating system. For more information, see ossutil command reference.

• Number of objects

  Without --delete, the number of objects is unlimited. With --delete, you can synchronize up to 1 million objects at a time. Exceeding this limit returns the over max sync numbers 1000000. error.

• Differences from the cp command

  • The sync command recursively traverses all objects and subdirectories by default. The cp command requires the -r option for recursive operations.

  • The sync command supports --backup-dir to save destination-only files to a backup folder. The cp command does not support --backup-dir.

  • The sync command does not support --version-id, so it cannot synchronize previous versions of objects. The cp command supports --version-id.

  The sync and cp commands share all other options. cp (Download files).

Command format

ossutil sync cloud_url  file_url
[-f --force]
[-u --update]
[--maxdownspeed <value>]
[--delete]
[--backup-dir <value>]
[--enable-symlink-dir]
[--disable-all-symlink]
[--disable-ignore-error]
[--only-current-dir]
[--output-dir <value>]
[--bigfile-threshold <value>]
[--part-size <value>]
[--checkpoint-dir <value>]
[--range <value>]
[--encoding-type <value>]
[--snapshot-path <value>]
[--include <value>]
[--exclude <value>]
[--disable-crc64]
[--payer <value>]
[-j, --job <value>]
[--parallel <value>]
[--retry-times <value>]

Parameters and options:

Verify sync results

Verify that all objects were transferred correctly:

• Check file count

  Compare object counts between OSS and local directories:

  # Count files in the local directory.
  ls -1 /local/dir/ | wc -l
  
  # Count objects in the OSS directory.  
  ossutil ls oss://bucket/dir/ -r | grep "^oss://" | wc -l

• List synchronized objects

  Review the synchronized objects:

  # List local files with sizes.
  ls -lh /local/dir/
  
  # Compare with the OSS directory.
  ossutil ls oss://bucket/dir/ -r

Examples

The following examples use this file structure. The OSS bucket localdir folder contains a.txt, b.txt, and subdirectory C. The local destdir folder contains d.txt:

examplebucket           local root directory
└── localdir/             └── destdir/
       ├── a.txt                └── d.txt
       ├── b.txt
       └── C/

• Synchronize the localdir folder from OSS to your local computer

  ossutil sync  oss://examplebucket/localdir/  destdir/ 

  Result: a.txt, b.txt, and subdirectory C are added to the local destdir folder.

  examplebucket           local root directory
  └── localdir/             └── destdir/
         ├── a.txt                ├── a.txt 
         ├── b.txt                ├── b.txt
         └── C/                   ├── d.txt
                                     └── C/ 

• Synchronize with --backup-dir: sync localdir from examplebucket to local destdir, and move destination-only files to the backup folder.

  ossutil sync oss://examplebucket/localdir/  destdir/  --delete  --backup-dir backup/

  Result: the localdir folder is synchronized. Files in destdir but not in the source are moved to the backup folder. The local destdir contains only a.txt, b.txt, and subdirectory C. The original d.txt is moved to the backup folder.

  examplebucket              local root directory
  └── localdir/               ├── destdir/
         ├── a.txt            │     ├── a.txt 
         ├── b.txt            │     ├── b.txt
         └── C/               │     └── C/                             
                                 └── backup/
                                        └──d.txt

• Successful synchronization output:

  Succeed: Total num: 2, size: 750,081. OK num: 2(upload 2 files).
  
  average speed 1641000(byte/s)

Common options

To access a bucket in a different region, use -e to specify the endpoint. To access a bucket owned by a different Alibaba Cloud account, use -i for the AccessKey ID and -k for the AccessKey secret.

ossutil sync oss://examplebucket/srcfolder/  examplefolder/ -e oss-cn-shanghai.aliyuncs.com -i yourAccessKeyID -k yourAccessKeySecret

Common options.