how to use oss go sdk v2 to upload files - Object Storage Service

This topic describes how to use the Uploader module of Go SDK V2 to upload files.

Usage notes

The sample code in this topic uses the region ID cn-hangzhou for the China (Hangzhou) region. By default, a public endpoint is used to access resources in a bucket. If you want to access bucket resources using other Alibaba Cloud services in the same region, use the internal endpoint. For more information about OSS regions and endpoints, see Regions and endpoints.
In this topic, access credentials are retrieved from environment variables. For more information about how to configure access credentials, see Configure access credentials.
To upload a large file, you must have the oss:PutObject permission. For more information, see Attach a custom policy to a RAM user.

Method definition

Uploader introduction

The Uploader in Go SDK V2 simplifies file uploads by providing a high-level operation that hides the underlying implementation details.

The Uploader uses the multipart upload operation to split a large file or stream into multiple smaller parts and uploads the parts concurrently to improve performance.
The Uploader also provides the resumable upload feature. This feature records the status of completed parts during the upload process. If the file upload fails due to issues such as network interruptions or program exceptions, you can resume the upload from the breakpoint record the next time you attempt to upload the file.

The following sample code shows how to use the Uploader:

type Uploader struct {
  ...
}

// Creates a new uploader.
func (c *Client) NewUploader(optFns ...func(*UploaderOptions)) *Uploader 

// Uploads a file stream.
func (u *Uploader) UploadFrom(ctx context.Context, request *PutObjectRequest, body io.Reader, optFns ...func(*UploaderOptions)) (*UploadResult, error)

// Uploads a local file.
func (u *Uploader) UploadFile(ctx context.Context, request *PutObjectRequest, filePath string, optFns ...func(*UploaderOptions)) (*UploadResult, error)

Request parameters

Parameter	Type	Description
ctx	context.Context	The context of the request.
request	*PutObjectRequest	The request parameters for uploading an object. The parameters are the same as those of the PutObject operation. For more information, see PutObjectRequest.
body	io.Reader	The stream to upload. If the body parameter supports only the io.Reader type, you must buffer the data in the memory before you can upload the data. If the body parameter supports the io.Reader, io.Seeker, and io.ReaderAt types, you do not need to buffer the data in the memory.
filePath	string	The path of the local file.
optFns	...func(*UploaderOptions)	(Optional) Configuration options.

The following table describes the common parameters of UploaderOptions.

Parameter	Type	Description
PartSize	int64	The part size. The default value is 6 MiB.
ParallelNum	int	The number of concurrent upload tasks. The default value is 3. This parameter specifies the concurrency limit for a single call, not the global concurrency limit.
LeavePartsOnError	bool	Specifies whether to retain the uploaded parts when an upload fails. By default, the uploaded parts are not retained.
EnableCheckpoint	bool	Specifies whether to enable the resumable upload feature. By default, this feature is disabled. Note The EnableCheckpoint parameter is valid only for the UploadFile operation and is not supported by the UploadFrom operation.
CheckpointDir	string	The path where the checkpoint file is saved. Example: /local/dir/. This parameter is valid only when EnableCheckpoint is set to true.

When you instantiate an Uploader using NewUploader, you can specify configuration options to customize the upload behavior. These options can be set for the Uploader instance to apply to all uploads, or specified for each individual upload operation.

Set the configuration parameters for the Uploader

u := client.NewUploader(func(uo *oss.UploaderOptions) {
  uo.PartSize = 10 * 1024 * 1024
})

Set the configuration parameters for each upload request

request := &oss.PutObjectRequest{Bucket: oss.Ptr("bucket"), Key: oss.Ptr("key")}
result, err := u.UploadFile(context.TODO(), request, "/local/dir/example", func(uo *oss.UploaderOptions) {
  uo.PartSize = 10 * 1024 * 1024
})

Examples

You can use the following sample code to upload a local file to a bucket using the Uploader.

package main

import (
	"context"
	"flag"
	"log"

	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
)

// Define global variables.
var (
	region     string // The storage region.
	bucketName string // The bucket name.
	objectName string // The object name.
)

// The init function is used to initialize command-line parameters.
func init() {
	flag.StringVar(&region, "region", "", "The region in which the bucket is located.")
	flag.StringVar(&bucketName, "bucket", "", "The name of the bucket.")
	flag.StringVar(&objectName, "object", "", "The name of the source object.")
}

func main() {
	// Parse command-line parameters.
	flag.Parse()

	// Check whether the bucket name is empty.
	if len(bucketName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, bucket name required")
	}

	// Check whether the region is empty.
	if len(region) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, region required")
	}

	// Check whether the object name is empty.
	if len(objectName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, source object name required")
	}

	// Load the default configurations and set the credential provider and region.
	cfg := oss.LoadDefaultConfig().
		WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
		WithRegion(region)

	// Create an OSS client.
	client := oss.NewClient(cfg)

	// Create an uploader.
	u := client.NewUploader()

	// Define the local file path. You need to replace the path with the actual local file path and filename.
	localFile := "/Users/yourLocalPath/yourFileName"

	// Upload the file.
	result, err := u.UploadFile(context.TODO(),
		&oss.PutObjectRequest{
			Bucket: oss.Ptr(bucketName),
			Key:    oss.Ptr(objectName)},
		localFile)
	if err != nil {
		log.Fatalf("failed to upload file %v", err)
	}

	// Print the upload result.
	log.Printf("upload file result:%#v\n", result)
}

Common scenarios

Enable the resumable upload feature using the uploader

You can use the following sample code to enable the resumable upload feature by setting the configuration parameters of the Uploader.

package main

import (
	"context"
	"flag"
	"log"

	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
)

// Define global variables.
var (
	region     string // The storage region.
	bucketName string // The bucket name.
	objectName string // The object name.
)

// The init function is used to initialize command-line parameters.
func init() {
	flag.StringVar(&region, "region", "", "The region in which the bucket is located.")
	flag.StringVar(&bucketName, "bucket", "", "The name of the bucket.")
	flag.StringVar(&objectName, "object", "", "The name of the source object.")
}

func main() {
	// Parse command-line parameters.
	flag.Parse()

	// Check whether the bucket name is empty.
	if len(bucketName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, bucket name required")
	}

	// Check whether the region is empty.
	if len(region) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, region required")
	}

	// Check whether the object name is empty.
	if len(objectName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, source object name required")
	}

	// Load the default configurations and set the credential provider and region.
	cfg := oss.LoadDefaultConfig().
		WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
		WithRegion(region)

	// Create an OSS client.
	client := oss.NewClient(cfg)

	// Create an uploader and enable the resumable upload feature.
	u := client.NewUploader(func(uo *oss.UploaderOptions) {
		uo.CheckpointDir = "/Users/yourLocalPath/checkpoint/" // Specify the path to save the checkpoint file.
		uo.EnableCheckpoint = true        // Enable resumable upload.
	})

	// Define the local file path. You need to replace the path with the actual local file path and filename.
	localFile := "/Users/yourLocalPath/yourFileName"

	// Upload the file.
	result, err := u.UploadFile(context.TODO(),
		&oss.PutObjectRequest{
			Bucket: oss.Ptr(bucketName),
			Key:    oss.Ptr(objectName)},
		localFile)
	if err != nil {
		log.Fatalf("failed to upload file %v", err)
	}

	// Print the upload result.
	log.Printf("upload file result:%#v\n", result)
}

Use the uploader to upload a local file stream

You can use the following sample code to upload a local file stream using the Uploader.

package main

import (
	"context"
	"flag"
	"io"
	"log"
	"os"

	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
)

// Define global variables.
var (
	region     string // The storage region.
	bucketName string // The bucket name.
	objectName string // The object name.
)

// The init function is used to initialize command-line parameters.
func init() {
	flag.StringVar(&region, "region", "", "The region in which the bucket is located.")
	flag.StringVar(&bucketName, "bucket", "", "The name of the bucket.")
	flag.StringVar(&objectName, "object", "", "The name of the object.")
}

func main() {
	// Parse command-line parameters.
	flag.Parse()

	// Check whether the bucket name is empty.
	if len(bucketName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, bucket name required")
	}

	// Check whether the region is empty.
	if len(region) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, region required")
	}

	// Check whether the object name is empty.
	if len(objectName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, object name required")
	}

	// Load the default configurations and set the credential provider and region.
	cfg := oss.LoadDefaultConfig().
		WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
		WithRegion(region)

	// Create an OSS client.
	client := oss.NewClient(cfg)

	// Create an uploader.
	u := client.NewUploader()

	// Replace "/Users/yourLocalPath/yourFileName" with the actual file path and filename, open the local file, and create an io.Reader instance.
	file, err := os.Open("/Users/yourLocalPath/yourFileName")
	if err != nil {
		log.Fatalf("failed to open local file %v", err)
	}
	defer file.Close()

	var r io.Reader = file

	// Upload the file stream.
	result, err := u.UploadFrom(context.TODO(),
		&oss.PutObjectRequest{
			Bucket: oss.Ptr(bucketName),
			Key:    oss.Ptr(objectName),
		},
		r, // The file stream to upload.
	)

	if err != nil {
		log.Fatalf("failed to upload file stream %v", err)
	}

	// Print the result of the file stream upload.
	log.Printf("upload file stream, etag: %v\n", oss.ToString(result.ETag))
}

Use the uploader to set the part size and concurrency

You can use the following sample code to set the part size and concurrency by setting the configuration parameters of the Uploader.

package main

import (
	"context"
	"flag"
	"log"

	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
)

// Define global variables.
var (
	region     string // The storage region.
	bucketName string // The bucket name.
	objectName string // The object name.
)

// The init function is used to initialize command-line parameters.
func init() {
	flag.StringVar(&region, "region", "", "The region in which the bucket is located.")
	flag.StringVar(&bucketName, "bucket", "", "The name of the bucket.")
	flag.StringVar(&objectName, "object", "", "The name of the source object.")
}

func main() {
	// Parse command-line parameters.
	flag.Parse()

	// Check whether the bucket name is empty.
	if len(bucketName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, bucket name required")
	}

	// Check whether the region is empty.
	if len(region) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, region required")
	}

	// Check whether the object name is empty.
	if len(objectName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, source object name required")
	}

	// Load the default configurations and set the credential provider and region.
	cfg := oss.LoadDefaultConfig().
		WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
		WithRegion(region)

	// Create an OSS client.
	client := oss.NewClient(cfg)

	// Create an uploader.
	u := client.NewUploader(func(uo *oss.UploaderOptions) {
		uo.PartSize = 1024 * 1024 * 5     // Set the part size to 5 MB.
		uo.ParallelNum = 5                // Set the number of concurrent uploads to 5.
	})

	// Define the local file path. You need to replace the path with the actual local file path and filename.
	localFile := "/Users/yourLocalPath/yourFileName"

	// Upload the file.
	result, err := u.UploadFile(context.TODO(),
		&oss.PutObjectRequest{
			Bucket: oss.Ptr(bucketName),
			Key:    oss.Ptr(objectName)},
		localFile)
	if err != nil {
		log.Fatalf("failed to upload file %v", err)
	}

	// Print the upload result.
	log.Printf("upload file result:%#v\n", result)
}

Use the uploader to set upload callbacks

Use the following sample code to notify an application server after a file is uploaded.

package main

import (
	"context"
	"encoding/base64"
	"encoding/json"
	"flag"
	"log"

	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
)

// Define global variables.
var (
	region     string // The storage region.
	bucketName string // The bucket name.
	objectName string // The object name.
)

// The init function is used to initialize command-line parameters.
func init() {
	flag.StringVar(&region, "region", "", "The region in which the bucket is located.")
	flag.StringVar(&bucketName, "bucket", "", "The name of the bucket.")
	flag.StringVar(&objectName, "object", "", "The name of the source object.")
}

func main() {
	// Parse command-line parameters.
	flag.Parse()

	// Check whether the bucket name is empty.
	if len(bucketName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, bucket name required")
	}

	// Check whether the region is empty.
	if len(region) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, region required")
	}

	// Check whether the object name is empty.
	if len(objectName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, source object name required")
	}

	// Load the default configurations and set the credential provider and region.
	cfg := oss.LoadDefaultConfig().
		WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
		WithRegion(region)

	// Create an OSS client.
	client := oss.NewClient(cfg)

	// Create an uploader and enable the resumable upload feature.
	u := client.NewUploader(func(uo *oss.UploaderOptions) {
		uo.PartSize = 1024 * 1024 * 5 // Set the part size to 5 MB.
		uo.ParallelNum = 5            // Set the number of concurrent uploads to 5.
	})

	// Define the local file path. You need to replace the path with your actual local file path.
	localFile := "/Users/yourLocalPath/yourFileName"

	// Define the callback parameters.
	callbackMap := map[string]string{
		"callbackUrl":      "https://example.com:23450/callback",                                                        // Set the URL of the callback server. Example: https://example.com:23450.
		"callbackBody":     "bucket=${bucket}&object=${object}&size=${size}&my_var_1=${x:my_var1}&my_var_2=${x:my_var2}", // Set the callback request body.
		"callbackBodyType": "application/x-www-form-urlencoded",                                                          // Set the callback request body type.
	}

	// Convert the callback parameters to JSON and then perform Base64 encoding to pass them as callback parameters.
	callbackStr, err := json.Marshal(callbackMap)
	if err != nil {
		log.Fatalf("failed to marshal callback map: %v", err)
	}
	callbackBase64 := base64.StdEncoding.EncodeToString(callbackStr)

	// Define custom callback parameters.
	callbackVarMap := map[string]string{}
	callbackVarMap["x:my_var1"] = "thi is var 1"
	callbackVarMap["x:my_var2"] = "thi is var 2"

	// Convert the callback parameters to JSON and then perform Base64 encoding to pass them as callback parameters.
	callbackVarStr, err := json.Marshal(callbackVarMap)
	if err != nil {
		log.Fatalf("failed to marshal callback var: %v", err)
	}
	callbackVarBase64 := base64.StdEncoding.EncodeToString(callbackVarStr)

	// Upload the large file.
	result, err := u.UploadFile(context.TODO(),
		&oss.PutObjectRequest{
			Bucket:      oss.Ptr(bucketName),
			Key:         oss.Ptr(objectName),
			Callback:    oss.Ptr(callbackBase64),    // Specify the callback parameters.
			CallbackVar: oss.Ptr(callbackVarBase64), // Specify the custom callback parameters.
		},
		localFile)
	if err != nil {
		log.Fatalf("failed to upload file %v", err)
	}

	// Print the upload result.
	log.Printf("upload file result:%#v\n", result)
}

Use the uploader to display a progress bar chart

package main

import (
	"context"
	"flag"
	"fmt"
	"log"

	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
)

// Define global variables.
var (
	region     string // The storage region.
	bucketName string // The bucket name.
	objectName string // The object name.
)

// The init function is used to initialize command-line parameters.
func init() {
	flag.StringVar(&region, "region", "", "The region in which the bucket is located.")
	flag.StringVar(&bucketName, "bucket", "", "The name of the bucket.")
	flag.StringVar(&objectName, "object", "", "The name of the source object.")
}

func main() {
	// Parse command-line parameters.
	flag.Parse()

	// Check whether the bucket name is empty.
	if len(bucketName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, bucket name required")
	}

	// Check whether the region is empty.
	if len(region) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, region required")
	}

	// Check whether the object name is empty.
	if len(objectName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, source object name required")
	}

	// Load the default configurations and set the credential provider and region.
	cfg := oss.LoadDefaultConfig().
		WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
		WithRegion(region)

	// Create an OSS client.
	client := oss.NewClient(cfg)

	// Create an uploader.
	u := client.NewUploader(func(uo *oss.UploaderOptions) {
		uo.PartSize = 1024 * 1024 * 5 // Set the part size to 5 MB.
		uo.ParallelNum = 3            // Set the number of concurrent uploads to 3.
	})

	// Replace the path with your actual local file path.
	localFile := "/Users/yourLocalPath/yourFileName"

	// Upload the large file.
	result, err := u.UploadFile(context.TODO(),
		&oss.PutObjectRequest{
			Bucket: oss.Ptr(bucketName),
			Key:    oss.Ptr(objectName),
			ProgressFn: func(increment, transferred, total int64) {
				fmt.Printf("increment:%v, transferred:%v, total:%v\n", increment, transferred, total)
			}, // The callback function for progress. This function is used to display the upload progress.
		},
		localFile)
	if err != nil {
		log.Fatalf("failed to upload file %v", err)
	}

	// Print the upload result.
	log.Printf("upload file result:%#v\n", result)
}

References

For more information about the Uploader, see Developer Guide.
For more information about the API operations that you can use to upload large files, see UploadFrom and UploadFile.