How to download files by using OSS SDK for Go V2 - Object Storage Service

This topic describes how to download files using the Downloader module, a new feature in Go SDK V2.

Usage notes

The sample code in this topic uses the region ID cn-hangzhou of the China (Hangzhou) region. By default, a public endpoint is used. If you want to access OSS from other Alibaba Cloud products in the same region, use an internal endpoint. For more information about the mappings between OSS regions and endpoints, see Regions and endpoints.
The examples in this topic read access credentials from environment variables. For more information about how to configure access credentials, see Configure access credentials.
To download files, you must have the oss:GetObject permission. For more information, see Attach a custom policy to a RAM user.

Method definition

Downloader features

The Downloader is a new feature in Go SDK V2 that provides a universal download operation. It abstracts the implementation details of the underlying operations to provide a convenient way to download files.

The Downloader uses range download to automatically split a file into smaller parts and downloads these parts concurrently to improve download performance.
The Downloader also provides a resumable download feature. During the download process, the status of completed parts is recorded. If a file download fails due to issues such as network interruptions or unexpected program exits, you can resume the download from the breakpoint record.

The common methods of the Downloader are as follows:

type Downloader struct {
  ...
}

// Creates a new downloader.
func (c *Client) NewDownloader(optFns ...func(*DownloaderOptions)) *Downloader

// Downloads a file.
func (d *Downloader) DownloadFile(ctx context.Context, request *GetObjectRequest, filePath string, optFns ...func(*DownloaderOptions)) (result *DownloadResult, err error)

Request parameters

Parameter	Type	Description
ctx	context.Context	The context of the request. You can use this parameter to set the total timeout period for the request.
request	*GetObjectRequest	The request parameters for the specific API operation. For more information, see GetObjectRequest.
filePath	string	The path of the local file.
optFns	...func(*DownloaderOptions)	(Optional) The configuration options.

The following table describes the common parameters of DownloaderOption.

Parameter	Type	Description
PartSize	int64	The part size. The default value is 6 MiB.
ParallelNum	int	The number of concurrent download tasks. The default value is 3. This parameter specifies the concurrency limit for a single call, not the global concurrency limit.
EnableCheckpoint	bool	Specifies whether to record resumable download information. By default, this feature is disabled.
CheckpointDir	string	The path where the record file is saved. Example: /local/dir/. This parameter takes effect only when EnableCheckpoint is set to true.
VerifyData	bool	Specifies whether to verify the CRC-64 value of the downloaded data when the download is resumed. By default, the CRC-64 value is not verified. This parameter takes effect only when EnableCheckpoint is set to true.
UseTempFile	bool	Specifies whether to use a temporary file for the download. By default, a temporary file is used. The file is first downloaded to a temporary file. After the download is successful, the temporary file is renamed to the object file.

When you instantiate a Downloader instance, you can specify configuration options to customize its default download behavior. You can also specify configuration options for each download operation to customize the behavior for a specific download.

Set the configuration parameters for the Downloader:

d := client.NewDownloader(func(do *oss.DownloaderOptions) {
  do.PartSize = 10 * 1024 * 1024
})

Set the configuration parameters for each download request:

request := &oss.GetObjectRequest{Bucket: oss.Ptr("bucket"), Key: oss.Ptr("key")}
d.DownloadFile(context.TODO(), request, "/local/dir/example", func(do *oss.DownloaderOptions) {
  do.PartSize = 10 * 1024 * 1024
})

Sample code

You can use the following code to download a file from a bucket to a local device.

package main

import (
	"context"
	"flag"
	"log"

	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
)

// Define global variables.
var (
	region     string // The storage region.
	bucketName string // The bucket name.
	objectName string // The object name.
)

// The init function is used to initialize command line parameters.
func init() {
	flag.StringVar(&region, "region", "", "The region in which the bucket is located.")
	flag.StringVar(&bucketName, "bucket", "", "The name of the bucket.")
	flag.StringVar(&objectName, "src-object", "", "The name of the source object.")
}

func main() {
	// Parse command line parameters.
	flag.Parse()

	// Check whether the bucket name is empty.
	if len(bucketName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, bucket name required")
	}

	// Check whether the region is empty.
	if len(region) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, region required")
	}

	// Check whether the source object name is empty.
	if len(objectName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, src object name required")
	}

	// Configure the OSS client.
	cfg := oss.LoadDefaultConfig().
		WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
		WithRegion(region)

	// Create an OSS client.
	client := oss.NewClient(cfg)

	// Create a downloader.
	d := client.NewDownloader()

	// Build a request to get the object.
	request := &oss.GetObjectRequest{
		Bucket: oss.Ptr(bucketName), // The bucket name.
		Key:    oss.Ptr(objectName), // The object name.
	}

	// Define the local file path.
	localFile := "local-file"

	// Execute the request to download the file.
	result, err := d.DownloadFile(context.TODO(), request, localFile)
	if err != nil {
		log.Fatalf("failed to download file %v", err)
	}

	// Print the success message.
	log.Printf("download file %s to local-file successfully, size: %d", objectName, result.Written)
}

Scenarios

Use the downloader to set the part size and concurrency

The following code shows how to set the part size and concurrency using the DownloaderOptions parameters.

package main

import (
	"context"
	"flag"
	"log"

	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
)

// Define global variables.
var (
	region     string // The storage region.
	bucketName string // The bucket name.
	objectName string // The object name.
)

// The init function is used to initialize command line parameters.
func init() {
	flag.StringVar(&region, "region", "", "The region in which the bucket is located.")
	flag.StringVar(&bucketName, "bucket", "", "The name of the bucket.")
	flag.StringVar(&objectName, "src-object", "", "The name of the source object.")
}

func main() {
	// Parse command line parameters.
	flag.Parse()

	// Check whether the bucket name is empty.
	if len(bucketName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, bucket name required")
	}

	// Check whether the region is empty.
	if len(region) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, region required")
	}

	// Check whether the source object name is empty.
	if len(objectName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, src object name required")
	}

	// Configure the OSS client.
	cfg := oss.LoadDefaultConfig().
		WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
		WithRegion(region)

	// Create an OSS client.
	client := oss.NewClient(cfg)

	// Create a downloader.
	d := client.NewDownloader()

	// Build a request to get the object.
	request := &oss.GetObjectRequest{
		Bucket: oss.Ptr(bucketName), // The bucket name.
		Key:    oss.Ptr(objectName), // The object name.
	}

	// Define the local file path.
	localFile := "local-file"

	// Set the configuration parameters for the downloader.
	downloaderOptions := func(do *oss.DownloaderOptions) {
		do.PartSize = 20 * 1024 * 1024 // Specify the part size as 20 MiB.
		do.ParallelNum = 6            // Specify the number of concurrent download tasks as 6.
	}

	// Execute the request to download the file.
	result, err := d.DownloadFile(context.TODO(), request, localFile, downloaderOptions)
	if err != nil {
		log.Fatalf("failed to download file %v", err)
	}

	// Print the success message.
	log.Printf("download file %s to local-file successfully, size: %d", objectName, result.Written)
}

Use the downloader to enable the resumable download feature

The following code shows how to enable the resumable download feature using the DownloaderOptions parameters.

package main

import (
	"context"
	"flag"
	"log"

	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
)

// Define global variables.
var (
	region     string // The storage region.
	bucketName string // The bucket name.
	objectName string // The object name.
)

// The init function is used to initialize command line parameters.
func init() {
	flag.StringVar(&region, "region", "", "The region in which the bucket is located.")
	flag.StringVar(&bucketName, "bucket", "", "The name of the bucket.")
	flag.StringVar(&objectName, "src-object", "", "The name of the source object.")
}

func main() {
	// Parse command line parameters.
	flag.Parse()

	// Check whether the bucket name is empty.
	if len(bucketName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, bucket name required")
	}

	// Check whether the region is empty.
	if len(region) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, region required")
	}

	// Check whether the source object name is empty.
	if len(objectName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, src object name required")
	}

	// Configure the OSS client.
	cfg := oss.LoadDefaultConfig().
		WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
		WithRegion(region)

	// Create an OSS client.
	client := oss.NewClient(cfg)

	// Create a downloader.
	d := client.NewDownloader()

	// Build a request to get the object.
	request := &oss.GetObjectRequest{
		Bucket: oss.Ptr(bucketName), // The bucket name.
		Key:    oss.Ptr(objectName), // The object name.
	}

	// Define the local file path.
	localFile := "local-file"

	// Set the downloader options.
	downloaderOptions := func(do *oss.DownloaderOptions) {
		do.EnableCheckpoint = true        // Enable resumable download information recording.
		do.CheckpointDir = "./checkpoint" // Specify the directory where the resumable download information is stored.
		do.UseTempFile = true             // Use a temporary file for the download.
	}

	// Execute the request to download the file.
	result, err := d.DownloadFile(context.TODO(), request, localFile, downloaderOptions)
	if err != nil {
		log.Fatalf("failed to download file %v", err)
	}

	// Print the success message.
	log.Printf("download file %s to local-file successfully, size: %d", objectName, result.Written)
}

References

For more information about the downloader, see Developer Guide.
For more information about the API operation for file download, see DownloadFile.