The Downloader splits objects into parts and downloads them concurrently, with optional resumable download support. It is the recommended approach for downloading large objects from OSS.
Prerequisites
Before you begin, ensure that you have:
An OSS bucket containing the object you want to download
The
oss:GetObjectpermission. For more information, see Attach a custom policy to a RAM userAccess credentials configured as environment variables. For more information, see Configure access credentials
Usage notes
The sample code uses the region ID
cn-hangzhouand a public endpoint by default. To access OSS from other Alibaba Cloud services in the same region, use an internal endpoint. For region-endpoint mappings, see Regions and endpoints.
How it works
The Downloader uses range download to split an object into parts of a configurable size (PartSize, default 6 MiB) and downloads them in parallel (ParallelNum, default 3 concurrent tasks). When resumable download is enabled, the Downloader records the status of completed parts to a checkpoint file. If the download fails due to a network interruption or unexpected program exit, the next call resumes from the last checkpoint rather than restarting from the beginning.
By default, the Downloader writes to a temporary file and renames it to the target path after the download completes successfully. This prevents partial files from appearing at the target path if the download fails.
API reference
Method signatures
type Downloader struct { ... }
// NewDownloader creates a Downloader. Pass functional options to override defaults.
func (c *Client) NewDownloader(optFns ...func(*DownloaderOptions)) *Downloader
// DownloadFile downloads an object to a local file.
func (d *Downloader) DownloadFile(ctx context.Context, request *GetObjectRequest, filePath string, optFns ...func(*DownloaderOptions)) (result *DownloadResult, err error)DownloadFile parameters
| Parameter | Type | Description |
|---|---|---|
ctx | context.Context | Request context. Use to set an overall timeout for the download. |
request | *GetObjectRequest | Object to download. See GetObjectRequest. |
filePath | string | Local path where the object is saved. |
optFns | ...func(*DownloaderOptions) | (Optional) Per-call configuration overrides. |
DownloaderOptions
| Parameter | Type | Default | Description |
|---|---|---|---|
PartSize | int64 | 6 MiB | Size of each download part. |
ParallelNum | int | 3 | Number of concurrent part downloads per DownloadFile call. This is per-call concurrency, not global. See Concurrency note below. |
EnableCheckpoint | bool | false | Enable resumable download. When true, the Downloader records completed parts to CheckpointDir. |
CheckpointDir | string | — | Directory for checkpoint files, e.g. ./checkpoint. Takes effect only when EnableCheckpoint is true. |
VerifyData | bool | false | Verify the CRC-64 checksum of downloaded data when resuming. Takes effect only when EnableCheckpoint is true. |
UseTempFile | bool | true | Write to a temporary file during download and rename it to filePath after success. |
Concurrency note
ParallelNum controls concurrency per DownloadFile call, not globally. If multiple goroutines call DownloadFile simultaneously on the same Downloader, the total concurrent part downloads equals (number of goroutines) x ParallelNum. For example, 4 goroutines with ParallelNum = 3 result in up to 12 concurrent downloads. Size ParallelNum accordingly to avoid exhausting application resources.
Download an object
package main
import (
"context"
"flag"
"log"
"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
)
var (
region string
bucketName string
objectName string
)
func init() {
flag.StringVar(®ion, "region", "", "The region in which the bucket is located.")
flag.StringVar(&bucketName, "bucket", "", "The name of the bucket.")
flag.StringVar(&objectName, "src-object", "", "The name of the source object.")
}
func main() {
flag.Parse()
if len(bucketName) == 0 {
flag.PrintDefaults()
log.Fatalf("invalid parameters, bucket name required")
}
if len(region) == 0 {
flag.PrintDefaults()
log.Fatalf("invalid parameters, region required")
}
if len(objectName) == 0 {
flag.PrintDefaults()
log.Fatalf("invalid parameters, src object name required")
}
// Load credentials from environment variables and set the region.
cfg := oss.LoadDefaultConfig().
WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
WithRegion(region)
client := oss.NewClient(cfg)
// Create a Downloader with default settings (6 MiB parts, 3 concurrent downloads).
d := client.NewDownloader()
request := &oss.GetObjectRequest{
Bucket: oss.Ptr(bucketName),
Key: oss.Ptr(objectName),
}
result, err := d.DownloadFile(context.TODO(), request, "local-file")
if err != nil {
log.Fatalf("failed to download file %v", err)
}
log.Printf("downloaded %s to local-file (%d bytes)", objectName, result.Written)
}Configure part size and concurrency
Pass options at the DownloadFile call level to override defaults for a specific download. These overrides do not affect other concurrent or subsequent calls.
// Create a Downloader with instance-level defaults.
d := client.NewDownloader(func(do *oss.DownloaderOptions) {
do.PartSize = 10 * 1024 * 1024 // 10 MiB default for all calls on this Downloader
})
request := &oss.GetObjectRequest{
Bucket: oss.Ptr(bucketName),
Key: oss.Ptr(objectName),
}
// Override options for this specific download only.
result, err := d.DownloadFile(context.TODO(), request, "local-file",
func(do *oss.DownloaderOptions) {
do.PartSize = 20 * 1024 * 1024 // 20 MiB for this call only
do.ParallelNum = 6 // 6 concurrent parts for this call only
},
)
if err != nil {
log.Fatalf("failed to download file %v", err)
}
log.Printf("downloaded %s to local-file (%d bytes)", objectName, result.Written)