Object Storage Service:Resumable download

How it works

The OSS SDK records download progress in a local checkpoint file. Each part is downloaded independently. If a part fails, the SDK resumes it from the recorded offset on the next attempt. After all parts complete successfully, the SDK assembles them into a single local file and deletes the checkpoint file.

Key behaviors:

• Checkpoint file: Created when a download is interrupted. Write permission is required for the checkpoint file path. If the checkpoint file is damaged or its checksum is modified, all parts must be re-downloaded from the beginning.

• ETag validation: If the object's ETag changes or parts are lost during download, re-download the entire object.

• Parallel parts: Configure the number of concurrent download threads to balance speed and resource usage.

Prerequisites

Before you begin, make sure that:

• OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are set with valid credentials

• The credentials include the oss:GetObject permission—see Attach a custom policy to a RAM user

• The local path for the checkpoint file has write permissions

• (Archive objects) The object is restored, or real-time access is enabled for the bucket—see Restore objects and Real-time access of Archive objects

• (Cold Archive or Deep Cold Archive objects) The object is restored—see Restore objects

Download an object using an OSS SDK

The following examples show how to perform resumable download using OSS SDKs. All examples enable checkpoint recording, configure a part size, and set the number of parallel download threads.

Note: The part sizes and thread counts in these examples are for illustration only. Tune these values based on your object size, available bandwidth, and application requirements.

To create an OSSClient instance using a custom domain name or Security Token Service (STS), see Configure OSSClient instances. To access OSS from another Alibaba Cloud service in the same region, use an internal endpoint instead of the public endpoint shown in these examples—see Regions and endpoints.

import com.aliyun.oss.ClientBuilderConfiguration;
import com.aliyun.oss.OSS;
import com.aliyun.oss.common.auth.*;
import com.aliyun.oss.OSSClientBuilder;
import com.aliyun.oss.OSSException;
import com.aliyun.oss.common.comm.SignVersion;
import com.aliyun.oss.model.*;

public class Demo {
    public static void main(String[] args) throws Exception {
        // Specify the endpoint for your bucket's region.
        String endpoint = "https://oss-cn-hangzhou.aliyuncs.com";
        // Load credentials from environment variables (OSS_ACCESS_KEY_ID, OSS_ACCESS_KEY_SECRET).
        EnvironmentVariableCredentialsProvider credentialsProvider = CredentialsProviderFactory.newEnvironmentVariableCredentialsProvider();
        String bucketName = "examplebucket";
        // Full object path, excluding the bucket name.
        String objectName = "exampledir/exampleobject.txt";
        String region = "cn-hangzhou";

        ClientBuilderConfiguration clientBuilderConfiguration = new ClientBuilderConfiguration();
        clientBuilderConfiguration.setSignatureVersion(SignVersion.V4);
        OSS ossClient = OSSClientBuilder.create()
            .endpoint(endpoint)
            .credentialsProvider(credentialsProvider)
            .clientConfiguration(clientBuilderConfiguration)
            .region(region)
            .build();

        try {
            DownloadFileRequest downloadFileRequest = new DownloadFileRequest(bucketName, objectName);
            // Local file path for the downloaded object.
            downloadFileRequest.setDownloadFile("D:\\localpath\\examplefile.txt");
            // Part size in bytes. Valid range: 100 KB-5 GB. Default: 100 KB.
            downloadFileRequest.setPartSize(1 * 1024 * 1024);
            // Number of concurrent download threads. Default: 1.
            downloadFileRequest.setTaskNum(10);
            // Enable checkpoint recording so interrupted downloads can resume.
            downloadFileRequest.setEnableCheckpoint(true);
            // Optional: specify the full path of the checkpoint file. Example: D:\\localpath\\examplefile.txt.dcp.
            // The checkpoint file is generated when the download is interrupted. If you want to resume the download task,
            // you must specify the full path of the checkpoint file. After the object is downloaded, the checkpoint file is deleted.
            //downloadFileRequest.setCheckpointFile("D:\\localpath\\examplefile.txt.dcp");

            DownloadFileResult downloadRes = ossClient.downloadFile(downloadFileRequest);
            // After the download completes, object metadata is returned.
            ObjectMetadata objectMetadata = downloadRes.getObjectMetadata();
            System.out.println(objectMetadata.getETag());
            System.out.println(objectMetadata.getLastModified());
            System.out.println(objectMetadata.getUserMetadata().get("meta"));
        } catch (OSSException oe) {
            System.out.println("OSS error. The request reached OSS but was rejected.");
            System.out.println("Error Message: " + oe.getErrorMessage());
            System.out.println("Error Code: " + oe.getErrorCode());
            System.out.println("Request ID: " + oe.getRequestId());
            System.out.println("Host ID: " + oe.getHostId());
        } catch (Throwable ce) {
            System.out.println("Client error. Could not communicate with OSS.");
            System.out.println("Error Message: " + ce.getMessage());
        } finally {
            if (ossClient != null) {
                ossClient.shutdown();
            }
        }
    }
}

import argparse
import alibabacloud_oss_v2 as oss

parser = argparse.ArgumentParser(description="download file sample")
parser.add_argument('--region', help='The region where the bucket is located.', required=True)
parser.add_argument('--bucket', help='The bucket name.', required=True)
parser.add_argument('--endpoint', help='Custom endpoint (optional).')
parser.add_argument('--key', help='The object key.', required=True)
parser.add_argument('--file_path', help='Local path to save the downloaded file.', required=True)

def main():
    args = parser.parse_args()

    # Load credentials from environment variables (OSS_ACCESS_KEY_ID, OSS_ACCESS_KEY_SECRET).
    credentials_provider = oss.credentials.EnvironmentVariableCredentialsProvider()

    cfg = oss.config.load_default()
    cfg.credentials_provider = credentials_provider
    cfg.region = args.region

    if args.endpoint is not None:
        cfg.endpoint = args.endpoint

    client = oss.Client(cfg)

    # Create a downloader with checkpoint and data verification enabled.
    downloader = client.downloader(
        use_temp_file=True,             # Write to a temporary file during download.
        enable_checkpoint=True,         # Enable resumable download.
        checkpoint_dir=args.file_path,  # Directory for the checkpoint file.
        verify_data=True                # Verify data integrity after download.
    )

    result = downloader.download_file(
        oss.GetObjectRequest(
            bucket=args.bucket,
            key=args.key,
        ),
        filepath=args.file_path
    )

    print(f'Written: {result.written} bytes')

if __name__ == "__main__":
    main()

package main

import (
    "context"
    "flag"
    "log"

    "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
    "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
)

var (
    region     string
    bucketName string
    objectName string
)

func init() {
    flag.StringVar(&region, "region", "", "The region where the bucket is located.")
    flag.StringVar(&bucketName, "bucket", "", "The bucket name.")
    flag.StringVar(&objectName, "src-object", "", "The object key.")
}

func main() {
    flag.Parse()

    if len(bucketName) == 0 {
        flag.PrintDefaults()
        log.Fatalf("invalid parameters, bucket name required")
    }
    if len(region) == 0 {
        flag.PrintDefaults()
        log.Fatalf("invalid parameters, region required")
    }
    if len(objectName) == 0 {
        flag.PrintDefaults()
        log.Fatalf("invalid parameters, src object name required")
    }

    // Load credentials from environment variables (OSS_ACCESS_KEY_ID, OSS_ACCESS_KEY_SECRET).
    cfg := oss.LoadDefaultConfig().
        WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
        WithRegion(region)

    client := oss.NewClient(cfg)
    d := client.NewDownloader()

    request := &oss.GetObjectRequest{
        Bucket: oss.Ptr(bucketName),
        Key:    oss.Ptr(objectName),
    }

    localFile := "local-file"

    // Configure checkpoint and temp file options.
    downloaderOptions := func(do *oss.DownloaderOptions) {
        do.EnableCheckpoint = true         // Enable checkpoint recording.
        do.CheckpointDir = "./checkpoint"  // Directory for the checkpoint file.
        do.UseTempFile = true              // Write to a temporary file during download.
    }

    result, err := d.DownloadFile(context.TODO(), request, localFile, downloaderOptions)
    if err != nil {
        log.Fatalf("failed to download file: %v", err)
    }

    log.Printf("Downloaded %s to %s successfully. Size: %d bytes", objectName, localFile, result.Written)
}

using Aliyun.OSS;
using Aliyun.OSS.Common;

// Specify the endpoint for your bucket's region.
var endpoint = "https://oss-cn-hangzhou.aliyuncs.com";
// Load credentials from environment variables (OSS_ACCESS_KEY_ID, OSS_ACCESS_KEY_SECRET).
var accessKeyId = Environment.GetEnvironmentVariable("OSS_ACCESS_KEY_ID");
var accessKeySecret = Environment.GetEnvironmentVariable("OSS_ACCESS_KEY_SECRET");
var bucketName = "examplebucket";
// Full object path, excluding the bucket name.
var objectName = "exampleobject.txt";
// Local path for the downloaded file. If a file with the same name exists, it is overwritten.
var downloadFilename = "D:\\localpath\\examplefile.txt";
// Path for the checkpoint file. Set to null to disable resumable download;
// the object will be re-downloaded from the start on failure.
var checkpointDir = "D:\\localpath\\examplefile.txt.dcp";
const string region = "cn-hangzhou";

var conf = new ClientConfiguration();
conf.SignatureVersion = SignatureVersion.V4;

var client = new OssClient(endpoint, accessKeyId, accessKeySecret, conf);
client.SetRegion(region);

try
{
    DownloadObjectRequest request = new DownloadObjectRequest(bucketName, objectName, downloadFilename)
    {
        // Part size in bytes.
        PartSize = 8 * 1024 * 1024,
        // Number of concurrent download threads.
        ParallelThreadCount = 3,
        // Checkpoint file path. Set to null to disable resumable download.
        CheckpointDir = checkpointDir,
    };
    client.ResumableDownloadObject(request);
    Console.WriteLine("Resumable download succeeded: {0}", objectName);
}
catch (OssException ex)
{
    Console.WriteLine("Failed with error code: {0}; Error info: {1}. \nRequestID: {2}\tHostID: {3}",
        ex.ErrorCode, ex.Message, ex.RequestId, ex.HostId);
}
catch (Exception ex)
{
    Console.WriteLine("Failed with error info: {0}", ex.Message);
}

#include <alibabacloud/oss/OssClient.h>
using namespace AlibabaCloud::OSS;

int main(void)
{
    // Specify the endpoint for your bucket's region.
    std::string Endpoint = "https://oss-cn-hangzhou.aliyuncs.com";
    std::string Region = "cn-hangzhou";
    std::string BucketName = "examplebucket";
    // Full object path, excluding the bucket name.
    std::string ObjectName = "exampledir/exampleobject.txt";
    // Local file path for the downloaded object.
    std::string DownloadFilePath = "D:\\localpath\\examplefile.txt";
    // Directory for the checkpoint file. Make sure this directory exists.
    // The checkpoint file is deleted automatically after a successful download.
    std::string CheckpointFilePath = "D:\\localpath";

    InitializeSdk();

    ClientConfiguration conf;
    conf.signatureVersion = SignatureVersionType::V4;
    // Load credentials from environment variables (OSS_ACCESS_KEY_ID, OSS_ACCESS_KEY_SECRET).
    auto credentialsProvider = std::make_shared<EnvironmentVariableCredentialsProvider>();
    OssClient client(Endpoint, credentialsProvider, conf);
    client.SetRegion(Region);

    DownloadObjectRequest request(BucketName, ObjectName, DownloadFilePath, CheckpointFilePath);
    auto outcome = client.ResumableDownloadObject(request);

    if (!outcome.isSuccess()) {
        std::cout << "ResumableDownloadObject failed"
            << ", code: " << outcome.error().Code()
            << ", message: " << outcome.error().Message()
            << ", requestId: " << outcome.error().RequestId() << std::endl;
        return -1;
    }

    ShutdownSdk();
    return 0;
}

#include "oss_api.h"
#include "aos_http_io.h"

const char *endpoint = "https://oss-cn-hangzhou.aliyuncs.com";
const char *bucket_name = "examplebucket";
// Full object path, excluding the bucket name.
const char *object_name = "exampledir/exampleobject.txt";
const char *local_filename = "yourLocalFilename";
const char *region = "cn-hangzhou";

void init_options(oss_request_options_t *options)
{
    options->config = oss_config_create(options->pool);
    aos_str_set(&options->config->endpoint, endpoint);
    // Load credentials from environment variables (OSS_ACCESS_KEY_ID, OSS_ACCESS_KEY_SECRET).
    aos_str_set(&options->config->access_key_id, getenv("OSS_ACCESS_KEY_ID"));
    aos_str_set(&options->config->access_key_secret, getenv("OSS_ACCESS_KEY_SECRET"));
    aos_str_set(&options->config->region, region);
    options->config->signature_version = 4;
    options->config->is_cname = 0;
    options->ctl = aos_http_controller_create(options->pool, 0);
}

int main(int argc, char *argv[])
{
    if (aos_http_io_initialize(NULL, 0) != AOSE_OK) {
        exit(1);
    }

    aos_pool_t *pool;
    aos_pool_create(&pool, NULL);
    oss_request_options_t *oss_client_options;
    oss_client_options = oss_request_options_create(pool);
    init_options(oss_client_options);

    aos_string_t bucket;
    aos_string_t object;
    aos_string_t file;
    aos_table_t *headers = NULL;
    aos_table_t *resp_headers = NULL;
    aos_status_t *resp_status = NULL;
    oss_resumable_clt_params_t *clt_params;

    aos_str_set(&bucket, bucket_name);
    aos_str_set(&object, object_name);
    aos_str_set(&file, local_filename);

    // Parameters: pool, part size (100 KB), thread count (3), enable checkpoint, checkpoint dir.
    clt_params = oss_create_resumable_clt_params_content(pool, 1024 * 100, 3, AOS_TRUE, NULL);
    resp_status = oss_resumable_download_file(oss_client_options, &bucket, &object, &file,
                                              headers, NULL, clt_params, NULL, &resp_headers);
    if (aos_status_is_ok(resp_status)) {
        printf("download succeeded\n");
    } else {
        printf("download failed\n");
    }

    aos_pool_destroy(pool);
    aos_http_io_deinitialize();
    return 0;
}

#import "DownloadService.h"
#import "OSSTestMacros.h"

@implementation DownloadRequest

@end

@implementation Checkpoint

- (instancetype)copyWithZone:(NSZone *)zone {
    Checkpoint *other = [[[self class] allocWithZone:zone] init];
    other.etag = self.etag;
    other.totalExpectedLength = self.totalExpectedLength;
    return other;
}

@end


@interface DownloadService()<NSURLSessionTaskDelegate, NSURLSessionDataDelegate>

@property (nonatomic, strong) NSURLSession *session;
@property (nonatomic, strong) NSURLSessionDataTask *dataTask;
@property (nonatomic, copy) DownloadFailureBlock failure;
@property (nonatomic, copy) DownloadSuccessBlock success;
@property (nonatomic, copy) DownloadProgressBlock progress;
@property (nonatomic, copy) Checkpoint *checkpoint;
@property (nonatomic, copy) NSString *requestURLString;
@property (nonatomic, copy) NSString *headURLString;
@property (nonatomic, copy) NSString *targetPath;
@property (nonatomic, assign) unsigned long long totalReceivedContentLength;
@property (nonatomic, strong) dispatch_semaphore_t semaphore;

@end

@implementation DownloadService

- (instancetype)init
{
    self = [super init];
    if (self) {
        NSURLSessionConfiguration *conf = [NSURLSessionConfiguration defaultSessionConfiguration];
        conf.timeoutIntervalForRequest = 15;

        NSOperationQueue *processQueue = [NSOperationQueue new];
        _session = [NSURLSession sessionWithConfiguration:conf delegate:self delegateQueue:processQueue];
        _semaphore = dispatch_semaphore_create(0);
        _checkpoint = [[Checkpoint alloc] init];
    }
    return self;
}

+ (instancetype)downloadServiceWithRequest:(DownloadRequest *)request {
    DownloadService *service = [[DownloadService alloc] init];
    if (service) {
        service.failure = request.failure;
        service.success = request.success;
        service.requestURLString = request.sourceURLString;
        service.headURLString = request.headURLString;
        service.targetPath = request.downloadFilePath;
        service.progress = request.downloadProgress;
        if (request.checkpoint) {
            service.checkpoint = request.checkpoint;
        }
    }
    return service;
}

/**
 * Send a HEAD request to get object metadata.
 * Compares the server ETag with the local checkpoint ETag to determine whether the download can resume.
 */
- (BOOL)getFileInfo {
    __block BOOL resumable = NO;
    NSURL *url = [NSURL URLWithString:self.headURLString];
    NSMutableURLRequest *request = [[NSMutableURLRequest alloc]initWithURL:url];
    [request setHTTPMethod:@"HEAD"];
    NSURLSessionDataTask *task = [[NSURLSession sharedSession] dataTaskWithRequest:request completionHandler:^(NSData * _Nullable data, NSURLResponse * _Nullable response, NSError * _Nullable error) {
        if (error) {
            NSLog(@"Failed to get object metadata: %@", error);
        } else {
            NSHTTPURLResponse *httpResponse = (NSHTTPURLResponse *)response;
            NSString *etag = [httpResponse.allHeaderFields objectForKey:@"Etag"];
            // Resume only if the server ETag matches the checkpoint ETag.
            resumable = [self.checkpoint.etag isEqualToString:etag];
        }
        dispatch_semaphore_signal(self.semaphore);
    }];
    [task resume];

    dispatch_semaphore_wait(self.semaphore, DISPATCH_TIME_FOREVER);
    return resumable;
}

/**
 * Return the size of the local partial file (used to calculate the resume offset).
 */
- (unsigned long long)fileSizeAtPath:(NSString *)filePath {
    unsigned long long fileSize = 0;
    NSFileManager *dfm = [NSFileManager defaultManager];
    if ([dfm fileExistsAtPath:filePath]) {
        NSError *error = nil;
        NSDictionary *attributes = [dfm attributesOfItemAtPath:filePath error:&error];
        if (!error && attributes) {
            fileSize = attributes.fileSize;
        } else if (error) {
            NSLog(@"error: %@", error);
        }
    }
    return fileSize;
}

- (void)resume {
    NSURL *url = [NSURL URLWithString:self.requestURLString];
    NSMutableURLRequest *request = [[NSMutableURLRequest alloc]initWithURL:url];
    [request setHTTPMethod:@"GET"];

    BOOL resumable = [self getFileInfo];
    if (resumable) {
        // Resume from the offset recorded in the local file.
        self.totalReceivedContentLength = [self fileSizeAtPath:self.targetPath];
        NSString *requestRange = [NSString stringWithFormat:@"bytes=%llu-", self.totalReceivedContentLength];
        [request setValue:requestRange forHTTPHeaderField:@"Range"];
    } else {
        self.totalReceivedContentLength = 0;
    }

    if (self.totalReceivedContentLength == 0) {
        [[NSFileManager defaultManager] createFileAtPath:self.targetPath contents:nil attributes:nil];
    }

    self.dataTask = [self.session dataTaskWithRequest:request];
    [self.dataTask resume];
}

- (void)pause {
    [self.dataTask cancel];
    self.dataTask = nil;
}

- (void)cancel {
    [self.dataTask cancel];
    self.dataTask = nil;
    [self removeFileAtPath:self.targetPath];
}

- (void)removeFileAtPath:(NSString *)filePath {
    NSError *error = nil;
    [[NSFileManager defaultManager] removeItemAtPath:self.targetPath error:&error];
    if (error) {
        NSLog(@"Failed to remove file: %@", error);
    }
}

#pragma mark - NSURLSessionDataDelegate

- (void)URLSession:(NSURLSession *)session task:(NSURLSessionTask *)task
didCompleteWithError:(nullable NSError *)error {
    NSHTTPURLResponse *httpResponse = (NSHTTPURLResponse *)task.response;
    if ([httpResponse isKindOfClass:[NSHTTPURLResponse class]]) {
        if (httpResponse.statusCode == 200) {
            self.checkpoint.etag = [[httpResponse allHeaderFields] objectForKey:@"Etag"];
            self.checkpoint.totalExpectedLength = httpResponse.expectedContentLength;
        } else if (httpResponse.statusCode == 206) {
            self.checkpoint.etag = [[httpResponse allHeaderFields] objectForKey:@"Etag"];
            self.checkpoint.totalExpectedLength = self.totalReceivedContentLength + httpResponse.expectedContentLength;
        }
    }

    if (error) {
        if (self.failure) {
            NSMutableDictionary *userInfo = [NSMutableDictionary dictionaryWithDictionary:error.userInfo];
            [userInfo oss_setObject:self.checkpoint forKey:@"checkpoint"];
            NSError *tError = [NSError errorWithDomain:error.domain code:error.code userInfo:userInfo];
            self.failure(tError);
        }
    } else if (self.success) {
        self.success(@{@"status": @"success"});
    }
}

- (void)URLSession:(NSURLSession *)session dataTask:(NSURLSessionDataTask *)dataTask
didReceiveResponse:(NSURLResponse *)response
completionHandler:(void (^)(NSURLSessionResponseDisposition))completionHandler
{
    NSHTTPURLResponse *httpResponse = (NSHTTPURLResponse *)dataTask.response;
    if ([httpResponse isKindOfClass:[NSHTTPURLResponse class]]) {
        if (httpResponse.statusCode == 200) {
            self.checkpoint.totalExpectedLength = httpResponse.expectedContentLength;
        } else if (httpResponse.statusCode == 206) {
            self.checkpoint.totalExpectedLength = self.totalReceivedContentLength + httpResponse.expectedContentLength;
        }
    }
    completionHandler(NSURLSessionResponseAllow);
}

// Append received data to the local file and update download progress.
- (void)URLSession:(NSURLSession *)session dataTask:(NSURLSessionDataTask *)dataTask
    didReceiveData:(NSData *)data {
    NSFileHandle *fileHandle = [NSFileHandle fileHandleForWritingAtPath:self.targetPath];
    [fileHandle seekToEndOfFile];
    [fileHandle writeData:data];
    [fileHandle closeFile];

    self.totalReceivedContentLength += data.length;
    if (self.progress) {
        self.progress(data.length, self.totalReceivedContentLength, self.checkpoint.totalExpectedLength);
    }
}

@end

require 'aliyun/oss'

client = Aliyun::OSS::Client.new(
  # Specify the endpoint for your bucket's region.
  endpoint: 'https://oss-cn-hangzhou.aliyuncs.com',
  # Load credentials from environment variables (OSS_ACCESS_KEY_ID, OSS_ACCESS_KEY_SECRET).
  access_key_id: ENV['OSS_ACCESS_KEY_ID'],
  access_key_secret: ENV['OSS_ACCESS_KEY_SECRET']
)

bucket = client.get_bucket('examplebucket')

# Basic resumable download with progress reporting.
bucket.resumable_download('exampledir/example.zip', '/tmp/example.zip') do |p|
  puts "Progress: #{p}"
end

# Resumable download with custom part size and checkpoint file path.
bucket.resumable_download(
  'exampledir/example.zip', '/tmp/example.zip',
  part_size: 100 * 1024,             # Part size: 100 KB.
  cpt_file: '/tmp/example.zip.cpt'   # Checkpoint file path.
) { |p|
  puts "Progress: #{p}"
}

For SDKs not listed here, see SDK overview.

What's next

• To download objects from a versioning-enabled bucket, see Manage objects in a versioning-enabled bucket.

• To let third-party users download objects from a private bucket, generate a signed URL or use STS temporary credentials—see Authorize third-party users to download objects.