Object Storage Service:Synchronize objects between OSS paths

Usage notes

• To synchronize objects between OSS buckets, you must have the oss:GetObject, oss:ListObjects, oss:PutObject, and oss:DeleteObject permissions. For more information, see Attach a custom policy to a RAM user.

• Binary name

  This topic provides sample command lines that work on the 64-bit Linux system. For other systems, replace ./ossutil64 in the commands with the corresponding binary name. For more information, see ossutil command reference.

• Number of local files to synchronize

  When you run the sync command without including the --delete option, the number of local files that you can synchronize at a time is unlimited. If you include the --delete option, you can synchronize up to 1 million local files at a time. If the number of local files that you want to synchronize exceeds 1 million, the over max sync numbers 1000000. error message is returned.

• Differences between sync and cp

  • The sync command recursively synchronizes all objects and subdirectories in the specified directory. The cp command recursively synchronizes objects only when the -r option is specified.

  • If you run the sync command to synchronize data to an OSS bucket, you can add the --delete option to the command to delete the following objects from the destination bucket: objects that exist in the destination bucket but do not exist in the source bucket. This way, only the synchronized objects are retained in the destination bucket. The cp command does not support the --delete option.

  • The sync command does not support the --version-id option. You cannot use the sync command to synchronize previous versions of objects in buckets for which versioning is enabled. The cp command supports the --version-id option.

  Except for the preceding differences, you can use the sync and cp commands in a similar manner. For more information about how to run the cp command, see Copy objects.

Limits

You cannot use the sync command to synchronize Cold Archive or Deep Cold Archive objects from a source bucket to a destination bucket.

Command syntax

./ossutil64 sync cloud_url cloud_url
[-f --force]
[-u --update]
[--delete]
[--enable-symlink-dir]
[--disable-all-symlink]
[--disable-ignore-error]
[--only-current-dir]
[--output-dir <value>]
[--bigfile-threshold <value>]
[--part-size <value>]
[--checkpoint-dir <value>]
[--encoding-type <value>]
[--include <value>]
[--exclude <value>]
[--meta <value>]
[--acl <value>]
[--maxupspeed <value>]
[--disable-crc64]
[--payer <value>]
[-j, --job <value>]
[--parallel <value>]
[--retry-times <value>]
[--tagging <value>]

The following table describes the parameters and options in the syntax.

Examples

examplebucket1          examplebucket2
├── exampledir/            └── destdir/
│      ├── a.txt               ├── c.txt
│      └── b.txt               └── e.txt
└── srcdir/     
        └── d.txt       

The following examples show how to run the sync command to synchronize objects between OSS paths in different scenarios.

• Synchronize the exampledir directory of examplebucket1 to the srcdir directory of the same bucket.

  ./ossutil64 sync oss://examplebucket1/exampledir/  oss://examplebucket1/srcdir/

  After you run the preceding command, all objects in the exampledir directory are synchronized to the srcdir directory. The following structure shows the objects and directories in examplebucket1 after synchronization:

  examplebucket1                    examplebucket1         
  ├── exampledir/                  ├── exampledir/
  │      ├── a.txt                 │      ├── a.txt 
  │      └── b.txt                 │      └── b.txt 
  └── srcdir/                      └── srcdir/    
          └── d.txt                        ├── a.txt
                                           ├── b.txt
                                           └── d.txt     

• Synchronize the exampledir directory of examplebucket1 to the destdir directory of examplebucket2.

  ./ossutil64 sync oss://examplebucket1/exampledir/ oss://examplebucket2/destdir/

  After you run the preceding command, all objects in the exampledir directory of examplebucket1 are synchronized to the destdir directory of examplebucket2. The following structure shows the objects and directories in examplebucket1 and examplebucket2 after synchronization:

  examplebucket1          examplebucket2
  ├── exampledir/            └── destdir/
  │      ├── a.txt               ├── a.txt  
  │      └── b.txt               ├── b.txt
  └── srcdir/                     ├── c.txt 
          └── d.txt               └── e.txt

• Synchronize all objects from examplebucket1 to examplebucket2 and use the --delete option to delete the objects that exist in examplebucket2 but do not exist in examplebucket1.

  ./ossutil64 sync oss://examplebucket1 oss://examplebucket2 --delete

  After you run the preceding command, the objects in examplebucket1 is synchronized to examplebucket2. The c.txt and e.txt objects that exist in examplebucket2 before synchronization are deleted. The following structure shows the objects and directories in examplebucket1 and examplebucket2 after synchronization:

  examplebucket1          examplebucket2
  ├── exampledir/            ├── exampledir/
  │      ├── a.txt           │      ├── a.txt  
  │      └── b.txt           │      └── b.txt
  └── srcdir/                └── srcdir/  
          └── d.txt                  └── d.txt                  

• If the preceding commands are run successfully, output similar to the following content is returned to indicate the number of synchronized objects, the sizes of synchronized objects, and the time consumed by the synchronization:

  Succeed: Total num: 2, size: 750,081. OK num: 2(upload 2 files).
  
  average speed 1641000(byte/s)

Common options

If you use ossutil to switch to a bucket that is located in another region, add the -e option to the command to specify the endpoint of the region in which the specified bucket is located. If you use ossutil to switch to a bucket that belongs to another Alibaba Cloud account, add the -i option to the command to specify the AccessKey ID of the specified account, and add the -k option to the command to specify the AccessKey secret of the specified account.

./ossutil64 sync oss://examplebucket/srcfolder/  oss://testbucket/examplefolder/ -e oss-cn-shanghai.aliyuncs.com -i LTAI4Fw2NbDUCV8zYUzA****  -k 67DLVBkH7EamOjy2W5RVAHUY9H****

For more information about other common options that you can use for the sync command, see Common options.