Use ossutil to perform resumable file transfer - Object Storage Service

In unstable network environments, connection interruptions can lead to file transfer failures. The resumable file transfer feature in ossutil 2.0 allows you to continue uploads or downloads from the point of interruption, enhancing transfer efficiency. By default, resumable file transfer is not enabled in ossutil 2.0. To use this feature, include the --checkpoint-dir parameter when executing the cp command to activate resumable file transfer.

Prerequisites

The OSS service is enabled.
A bucket is created.
The ossutil 2.0 command line tool is installed.

Resumable file transfer mechanism

Resumable file transfer splits a file into multiple segments for transfer. It allows resuming from the last stopped segment in the event of an interruption, ensuring data integrity and transfer efficiency.

Initiate transfer: You can use the --checkpoint-dir parameter to enable the resumable file transfer feature when transferring files.
Store information: Once the resumable file transfer feature is enabled, ossutil stores essential transfer progress details (including file name, size, upload id, and uploaded shards) in a .ucp file within the checkpoint-dir.
Recover from interruption: If the transfer is interrupted because of network issues or other reasons, you can restart the transfer. Ossutil will consult the .ucp file in the checkpoint-dir to determine which shards have been transferred and which need to continue.
Continue transfer: Ossutil resumes transferring the remaining shards based on the information about the shards already transferred.

Parameter description

Parameter	Description
--checkpoint-dir	The directory in which the checkpoint file is stored.

Examples

Upload a single file

The following example shows how to use the cp command in ossutil to upload the local file upload.rar to the specified bucket. Once the upload begins, a .ucp file is created in the your-checkpoint-dir directory to record upload progress.

ossutil cp upload.rar oss://bucket --checkpoint-dir D:\\your-checkpoint-dir\

If the upload is interrupted because of network issues, disk read/write problems, or other reasons, you might encounter the following error message:

Error: upload failed, upload id: 1F71C936CF7B423493BC7A664DB3FD82, cause: operation error UploadPart: Put "https://bucket.oss-cn-hangzhou.aliyuncs.com/upload.rar?partNumber=15&uploadId=1F71C936CF7B423493BC7A664DB3FD82": dial tcp: lookup bucket.oss-cn-hangzhou.aliyuncs.com: no such host

In such cases, you can resume the upload by rerunning the same upload command, utilizing the information in the .ucp file:

ossutil cp upload.rar oss://bucket --checkpoint-dir D:\\your-checkpoint-dir\

Once the transfer restarts, ossutil accesses the upload progress data in the checkpoint-dir directory and continues uploading the remaining shards:

Total 1 objects,6.833 GiB, Upload... done:(0 objects), 3.000%, avg 4.740 MiB/s

Batch upload files

This example illustrates how to batch upload files from a local local_directory to a specified bucket using the cp command in ossutil. During this process:

The -u parameter ensures only source files newer than the target files are uploaded.
The -r parameter enables recursive copying of all files within the folder and its subdirectories.

Progress information is stored in a .ucp file created in the your-checkpoint-dir directory during the upload.

 ossutil cp -u -r local_directory oss://bucket --checkpoint-dir D:\\your-checkpoint-dir\

If the upload is interrupted because of network issues, disk read/write problems, or other reasons, you may see the following error message:

Error occurs. See more information in file: /ossutil-2.0.6-beta.01091200-mac-arm64/ossutil_output/ossutil_report_20250122_115545_96481.report
FinishWithError: Total 2 files, 7836503730 B, Upload done:(0 objects) failed:(2 objects, 7836503730 B)

In such situations, rerun the same command to read the information from the .ucp file and continue the upload.

 ossutil cp -u -r local_directory oss://bucket --checkpoint-dir D:\\your-checkpoint-dir\

After resuming, ossutil will read the upload progress information from the checkpoint-dir directory and proceed with uploading the remaining shards:

Total 2 files,7.298 GiB, Upload... done:(0 objects), 1.645%, avg 17.521 MiB/s

Download a single file

The following example shows how to use the cp command in ossutil to download the download.rar file from a remote bucket to the current local directory with the name download.rar. Once the download begins, a .dcp file is created in the your-checkpoint-dir/.ossutil_checkpoint directory to record download progress.

ossutil cp oss://bucket/download.rar ./download.rar --checkpoint-dir D:\\your-checkpoint-dir\

Batch download files

The following example shows how to use the cp command in ossutil to download all files from the folder directory in a remote bucket to the current local local_directory directory. Once the download begins, a .dcp file is created in the your-checkpoint-dir/.ossutil_checkpoint directory to record download progress.

The -u parameter ensures only source files newer than the target files are uploaded.
The -r parameter enables recursive copying of all files within the folder and its subdirectories.

 ossutil cp -u -r oss://bucket/folder ./local_director --checkpoint-dir D:\\your-checkpoint-dir\