how to perform an append upload by using go sdk v2 - Object Storage Service

Append upload lets you append content directly to the end of an existing appendable object. This topic describes how to perform an append upload using the OSS SDK for Go.

Usage notes

The sample code in this topic uses the region ID cn-hangzhou for the China (Hangzhou) region. By default, the public Endpoint is used to access resources in a bucket. If you want to access resources in the bucket using other Alibaba Cloud services in the same region in which the bucket is located, you can use the internal Endpoint. For more information about OSS regions and endpoints, see Regions and endpoints.
In this topic, access credentials are read from environment variables. For more information about how to configure access credentials, see Configure access credentials.
If the file to which you want to append content does not exist, an appendable object is created when you call an append upload API operation.
If the file to which you want to append content exists:
- If the file is an appendable object and the specified append position is the same as the current length of the file, content is appended to the end of the file.
- If the file is an appendable object but the specified append position is not the same as the current length of the file, the PositionNotEqualToLength exception is thrown.
- If the file is not an appendable object, such as a normal object uploaded using simple upload, the ObjectNotAppendable exception is thrown.

Permissions

By default, an Alibaba Cloud account has full permissions. RAM users or RAM roles under an Alibaba Cloud account do not have any permissions by default. The Alibaba Cloud account or account administrator must grant operation permissions through RAM Policy or Bucket Policy.

API

Action

Definition

AppendObject

oss:PutObject

You can call this operation to upload an object by appending the object to an existing object.

oss:PutObjectTagging

When uploading an object by appending the object to an existing object, if you specify object tags through x-oss-tagging, this permission is required.

Method definition

For append upload scenarios, the OSS SDK for Go provides the new AppendFile operation to simulate file read and write behaviors for objects in a bucket. The following table describes the AppendFile and AppendObject operations.

Operation name

Description

Client.AppendObject

Performs an append upload. The final object can be up to 5 GiB in size.

Supports CRC-64 data integrity check (enabled by default).

Supports progress bars.

The request body is of the io.Reader type. If the io.Seeker type is supported, retransmission is available upon failure. This operation is not idempotent, and retransmission may fail.

Client.AppendFile

Provides the same capabilities as the Client.AppendObject operation.

Optimizes fault tolerance for retransmission failures.

Includes the AppendOnlyFile operation.

AppendOnlyFile.Write

AppendOnlyFile.WriteFrom

Premium append upload API: AppendFile

You can call the AppendFile operation to upload data by appending it. If the object does not exist, an appendable object is created. If the object exists but is not an appendable object, an error is returned.

The following sample code provides an example of how to define the AppendFile operation.

type AppendOnlyFile struct {
...
}

func (c *Client) AppendFile(ctx context.Context, bucket string, key string, optFns ...func(*AppendOptions)) (*AppendOnlyFile, error)

Request parameters

Parameter	Type	Description
ctx	context.Context	The context of the request.
bucket	string	The name of the bucket.
key	string	The name of the object.
optFns	...func(*AppendOptions)	Optional. The configuration options for appending a file.

The following table describes the parameters of AppendOptions.

Parameter	Type	Description
RequestPayer	*string	If the pay-by-requester mode is enabled, set this parameter to 'requester'.
CreateParameter	*AppendObjectRequest	Used to set the metadata of an object during the first upload. The metadata includes ContentType, Metadata, permissions, and storage class. For more information, see AppendObjectRequest.

Return values

Return value	Type	Description
file	*AppendOnlyFile	The instance of the appendable file. This parameter is valid only when err is nil. For more information, see AppendOnlyFile.
err	error	The status of opening the appendable file. If the operation fails, err is not nil.

The following table describes the AppendOnlyFile operation.

Operation name	Description
Close() error	Closes the file handle to release resources.
Write(b []byte) (int, error)	Writes the data in b to the data stream and returns the number of bytes written and any error that is encountered.
WriteFrom(r io.Reader) (int64, error)	Writes the data in r to the data stream and returns the number of bytes written and any error that is encountered.
Stat() (os.FileInfo, error)	Obtains the object information, including the object size, last modified time, and metadata.

Basic append upload API: AppendObject

func (c *Client) AppendObject(ctx context.Context, request *AppendObjectRequest, optFns ...func(*Options)) (*AppendObjectResult, error)

Request parameters

Parameter	Type	Description
ctx	context.Context	The context of the request, which can be used to set the total timeout period of the request.
request	*AppendObjectRequest	The request parameters for a specific API operation. For more information, see AppendObjectRequest.
optFns	...func(*Options)	Optional. The operation-level configuration parameters. For more information, see Options.

Return values

Return value	Type	Description
result	*AppendObjectResult	The return value of the API operation. This parameter is valid only when err is nil. For more information, see AppendObjectResult.
err	error	The status of the request. If the request fails, err is not nil.

Sample code

Use AppendFile to perform an append upload

package main

import (
	"context"
	"flag"
	"log"
	"os"

	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
)

// Define global variables.
var (
	region     string // The storage region.
	bucketName string // The bucket name.
	objectName string // The object name.
)

// The init function is used to initialize command-line parameters.
func init() {
	flag.StringVar(&region, "region", "", "The region in which the bucket is located.")
	flag.StringVar(&bucketName, "bucket", "", "The name of the bucket.")
	flag.StringVar(&objectName, "object", "", "The name of the object.")
}

func main() {
	// Parse command-line parameters.
	flag.Parse()

	// Check whether the bucket name is empty.
	if len(bucketName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, bucket name required")
	}

	// Check whether the storage region is empty.
	if len(region) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, region required")
	}

	// Check whether the object name is empty.
	if len(objectName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, object name required")
	}

	// Configure the OSS client.
	cfg := oss.LoadDefaultConfig().
		WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
		WithRegion(region)

	// Create an OSS client.
	client := oss.NewClient(cfg)

	// Create an instance of an appendable file.
	f, err := client.AppendFile(context.TODO(),
		bucketName,
		objectName,
		func(ao *oss.AppendOptions) {
			ao.CreateParameter = &oss.AppendObjectRequest{
				Acl: oss.ObjectACLPrivate, // Set the access permissions of the object to private.
				Metadata: map[string]string{
					"user": "jack", // Set the metadata of the object.
				},
				Tagging: oss.Ptr("key=value"), // Set the tags for the object.
			}
		})

	if err != nil {
		log.Fatalf("failed to append file %v", err)
	}
	defer f.Close() // Make sure that the data stream is closed when the function ends.

	// Open the local file example1.txt.
	lf, err := os.Open("/local/dir/example1.txt")
	if err != nil {
		log.Fatalf("failed to open local file %v", err)
	}

	// Append the content of example1.txt to the OSS object.
	_, err = f.WriteFrom(lf)
	if err != nil {
		log.Fatalf("failed to append file %v", err)
	}
	lf.Close() // Close the local file.

	// Open the local file example2.txt.
	lf, err = os.Open("/local/dir/example2.txt")
	if err != nil {
		log.Fatalf("failed to open local file %v", err)
	}

	// Append the content of example2.txt to the OSS object.
	_, err = f.WriteFrom(lf)
	if err != nil {
		log.Fatalf("failed to append file %v", err)
	}
	lf.Close() // Close the local file.

	// Read the content of the local file example3.txt.
	lb, err := os.ReadFile("/local/dir/example3.txt")
	if err != nil {
		log.Fatalf("failed to read local file %v", err)
	}

	// Append the content of example3.txt to the OSS object.
	_, err = f.Write(lb)
	if err != nil {
		log.Fatalf("failed to append file %v", err)
	}

	// Print the log information about the successful append upload.
	log.Printf("append file successfully")
}

Use AppendObject to perform an append upload

package main

import (
	"context"
	"flag"
	"log"
	"strings"

	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
)

// Define global variables.
var (
	region     string
	bucketName string
	objectName string
)

// The init function is used to initialize command-line parameters.
func init() {
	flag.StringVar(&region, "region", "", "The region in which the bucket is located.")
	flag.StringVar(&bucketName, "bucket", "", "The name of the bucket.")
	flag.StringVar(&objectName, "object", "", "The name of the object.")
}

func main() {
	// Parse command-line parameters.
	flag.Parse()

	// Define the initial position for the append upload.
	var (
		position = int64(0)
	)

	// Check whether the bucket name is empty.
	if len(bucketName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, bucket name required")
	}

	// Check whether the region is empty.
	if len(region) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, region required")
	}

	// Check whether the object name is empty.
	if len(objectName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, object name required")
	}

	// Load the default configurations and set the credential provider and region.
	cfg := oss.LoadDefaultConfig().
		WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
		WithRegion(region)

	// Create an OSS client.
	client := oss.NewClient(cfg)

	// Define the content to append.
	content := "hi append object"

	// Create an AppendObject request.
	request := &oss.AppendObjectRequest{
		Bucket:   oss.Ptr(bucketName),
		Key:      oss.Ptr(objectName),
		Position: oss.Ptr(position),
		Body:     strings.NewReader(content),
	}

	// Execute the AppendObject request and process the result.
	// The position for the first append upload is 0. The return value contains the position for the next append upload.
	result, err := client.AppendObject(context.TODO(), request)
	if err != nil {
		log.Fatalf("failed to append object %v", err)
	}

	// Create the second AppendObject request.
	request = &oss.AppendObjectRequest{
		Bucket:   oss.Ptr(bucketName),
		Key:      oss.Ptr(objectName),
		Position: oss.Ptr(result.NextPosition), // Obtain NextPosition from the return value of the first AppendObject request.
		Body:     strings.NewReader("hi append object"),
	}

	// Execute the second AppendObject request and process the result.
	result, err = client.AppendObject(context.TODO(), request)
	if err != nil {
		log.Fatalf("failed to append object %v", err)
	}

	log.Printf("append object result:%#v\n", result)
}

Common scenarios

Perform an append upload and display a progress bar

package main

import (
	"context"
	"flag"
	"fmt"
	"log"
	"strings"

	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
)

// Define global variables.
var (
	region     string
	bucketName string
	objectName string
)

// The init function is used to initialize command-line parameters.
func init() {
	flag.StringVar(&region, "region", "", "The region in which the bucket is located.")
	flag.StringVar(&bucketName, "bucket", "", "The name of the bucket.")
	flag.StringVar(&objectName, "object", "", "The name of the object.")
}

func main() {
	// Parse command-line parameters.
	flag.Parse()

	// Define the initial position for the append upload.
	var (
		position = int64(0)
	)

	// Check whether the bucket name is empty.
	if len(bucketName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, bucket name required")
	}

	// Check whether the region is empty.
	if len(region) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, region required")
	}

	// Check whether the object name is empty.
	if len(objectName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, object name required")
	}

	// Load the default configurations and set the credential provider and region.
	cfg := oss.LoadDefaultConfig().
		WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
		WithRegion(region)

	// Create an OSS client.
	client := oss.NewClient(cfg)

	// Define the content to append.
	content := "hi append object"

	// Create an AppendObject request.
	request := &oss.AppendObjectRequest{
		Bucket:   oss.Ptr(bucketName),
		Key:      oss.Ptr(objectName),
		Position: oss.Ptr(position),
		Body:     strings.NewReader(content),
		ProgressFn: func(increment, transferred, total int64) {
			fmt.Printf("increment:%v, transferred:%v, total:%v\n", increment, transferred, total)
		}, // The progress callback function, which is used to display the upload progress.
	}

	// Execute the first AppendObject request and process the result.
	// The position for the first append upload is 0. The return value contains the position for the next append upload.
	result, err := client.AppendObject(context.TODO(), request)
	if err != nil {
		log.Fatalf("failed to append object %v", err)
	}

	// Create the second AppendObject request.
	request = &oss.AppendObjectRequest{
		Bucket:   oss.Ptr(bucketName),
		Key:      oss.Ptr(objectName),
		Position: oss.Ptr(result.NextPosition), // Obtain NextPosition from the return value of the first AppendObject request.
		Body:     strings.NewReader("hi append object"),
		ProgressFn: func(increment, transferred, total int64) {
			fmt.Printf("increment:%v, transferred:%v, total:%v\n", increment, transferred, total)
		}, // The progress callback function, which is used to display the upload progress.
	}

	// Execute the second AppendObject request and process the result.
	result, err = client.AppendObject(context.TODO(), request)
	if err != nil {
		log.Fatalf("failed to append object %v", err)
	}

	// Print the versionId of the appended object.
	log.Printf("append object result:%#v\n", *result.VersionId)
}

References

For the complete sample code for append upload, see GitHub examples and Developer Guide.
For more information about the premium API operation for append upload, see AppendFile.
For more information about the basic API operation for append upload, see AppendObject.