Append upload objects using Go SDK V2 - Object Storage Service

Append upload lets you add content to the end of an existing appendable object. The final object can be up to 5 GiB.

The OSS SDK for Go V2 provides two APIs for append upload:

API	Description	When to use
`AppendFile` (recommended)	High-level API. Handles position tracking and retransmission failures automatically.	Most use cases, especially log streaming and progressive file building
`AppendObject`	Low-level API. Requires manual position tracking via `NextPosition`. Supports progress callbacks.	When you need fine-grained control over each append request

Prerequisites

Before you begin, ensure that you have:

The OSS SDK for Go V2 installed. See Installation
Access credentials configured as environment variables. See Configure access credentials
The oss:PutObject permission on the target bucket. If you set object tags during the upload, you also need oss:PutObjectTagging

Usage notes

Sample code in this topic uses the region ID cn-hangzhou (China (Hangzhou)). Replace it with your actual region ID. For the full list of regions and endpoints, see Regions and endpoints.
By default, the public endpoint is used. To access bucket resources from another Alibaba Cloud service in the same region, use the internal endpoint instead.
If the target object does not exist, the API creates a new appendable object.
If the target object exists:
- If it is an appendable object and the specified position equals the current object size, content is appended to the end.
- If it is an appendable object but the specified position does not equal the current object size, PositionNotEqualToLength is thrown.
- If it is not an appendable object (for example, an object uploaded with simple upload), ObjectNotAppendable is thrown.

Permissions

By default, an Alibaba Cloud account has full permissions. RAM users and RAM roles have no permissions by default. Grant permissions using RAM Policy or Bucket Policy.

API	Required action	Condition
AppendObject	`oss:PutObject`	Always required
AppendObject	`oss:PutObjectTagging`	Required only when setting object tags via `x-oss-tagging`

AppendFile (recommended)

AppendFile is the recommended API for append upload. It provides the same capabilities as AppendObject and handles position tracking and retransmission failures automatically, so you can write data in sequence without managing the append position yourself. If the object does not exist, an appendable object is created. If the object exists but is not appendable, an error is returned.

Method signature

func (c *Client) AppendFile(ctx context.Context, bucket string, key string, optFns ...func(*AppendOptions)) (*AppendOnlyFile, error)

Request parameters

Parameter	Type	Description
`ctx`	`context.Context`	The context of the request
`bucket`	`string`	The bucket name
`key`	`string`	The object key (object name)
`optFns`	`...func(*AppendOptions)`	(Optional) Configuration options

AppendOptions fields:

Field	Type	Description
`RequestPayer`	`*string`	Set to `"requester"` if pay-by-requester mode is enabled
`CreateParameter`	`*AppendObjectRequest`	Metadata to set on the first upload, including `ContentType`, `Metadata`, ACL, and storage class. See AppendObjectRequest

Return values

Value	Type	Description
`file`	`*AppendOnlyFile`	The appendable file handle. Valid only when `err` is `nil`. See AppendOnlyFile
`err`	`error`	`nil` on success

AppendOnlyFile methods:

Method	Description
`Close() error`	Closes the file handle and releases resources
`Write(b []byte) (int, error)`	Writes bytes from `b` to the data stream
`WriteFrom(r io.Reader) (int64, error)`	Writes data from an `io.Reader` to the data stream
`Stat() (os.FileInfo, error)`	Returns object information: size, last modified time, and metadata

AppendObject

AppendObject is the low-level API for append upload. Each call requires you to pass the current append position, which you get from NextPosition in the previous call's result. The first call always starts at position 0.

AppendObject supports CRC-64 data integrity verification (enabled by default) and progress callbacks. The request body is an io.Reader. If the body also implements io.Seeker, the SDK can retry on failure. Otherwise, retransmission is not available and the operation is not idempotent.

Method signature

func (c *Client) AppendObject(ctx context.Context, request *AppendObjectRequest, optFns ...func(*Options)) (*AppendObjectResult, error)

Request parameters

Parameter	Type	Description
`ctx`	`context.Context`	The context of the request. Used to set the total request timeout
`request`	`*AppendObjectRequest`	The append request parameters. See AppendObjectRequest
`optFns`	`...func(*Options)`	(Optional) Operation-level configuration. See Options

Return values

Value	Type	Description
`result`	`*AppendObjectResult`	The API result, including `NextPosition` for the next append call. Valid only when `err` is `nil`. See AppendObjectResult
`err`	`error`	`nil` on success

Sample code

Append multiple files using AppendFile

The following example opens an appendable file on OSS and sequentially appends the contents of three local files. WriteFrom accepts an io.Reader, and Write accepts a byte slice. Call Close when done to release the file handle.

package main

import (
	"context"
	"flag"
	"log"
	"os"

	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
)

var (
	region     string
	bucketName string
	objectName string
)

func init() {
	flag.StringVar(&region, "region", "", "The region in which the bucket is located.")
	flag.StringVar(&bucketName, "bucket", "", "The name of the bucket.")
	flag.StringVar(&objectName, "object", "", "The name of the object.")
}

func main() {
	flag.Parse()

	if len(bucketName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, bucket name required")
	}

	if len(region) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, region required")
	}

	if len(objectName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, object name required")
	}

	cfg := oss.LoadDefaultConfig().
		WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
		WithRegion(region)

	client := oss.NewClient(cfg)

	// Open the appendable file. If the object does not exist, it is created.
	// CreateParameter sets the ACL, metadata, and tags on the first write.
	f, err := client.AppendFile(context.TODO(),
		bucketName,
		objectName,
		func(ao *oss.AppendOptions) {
			ao.CreateParameter = &oss.AppendObjectRequest{
				Acl: oss.ObjectACLPrivate,
				Metadata: map[string]string{
					"user": "jack",
				},
				Tagging: oss.Ptr("key=value"),
			}
		})
	if err != nil {
		log.Fatalf("failed to append file %v", err)
	}
	defer f.Close()

	// Append example1.txt using WriteFrom (io.Reader).
	lf, err := os.Open("/local/dir/example1.txt")
	if err != nil {
		log.Fatalf("failed to open local file %v", err)
	}
	_, err = f.WriteFrom(lf)
	if err != nil {
		log.Fatalf("failed to append file %v", err)
	}
	lf.Close()

	// Append example2.txt using WriteFrom (io.Reader).
	lf, err = os.Open("/local/dir/example2.txt")
	if err != nil {
		log.Fatalf("failed to open local file %v", err)
	}
	_, err = f.WriteFrom(lf)
	if err != nil {
		log.Fatalf("failed to append file %v", err)
	}
	lf.Close()

	// Append example3.txt using Write ([]byte).
	lb, err := os.ReadFile("/local/dir/example3.txt")
	if err != nil {
		log.Fatalf("failed to read local file %v", err)
	}
	_, err = f.Write(lb)
	if err != nil {
		log.Fatalf("failed to append file %v", err)
	}

	log.Printf("append file successfully")
}

Append content using AppendObject

All AppendObject calls use a Position field to specify where to start writing. The first call sets Position to 0. Each subsequent call reads NextPosition from the previous result.

package main

import (
	"context"
	"flag"
	"log"
	"strings"

	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
)

var (
	region     string
	bucketName string
	objectName string
)

func init() {
	flag.StringVar(&region, "region", "", "The region in which the bucket is located.")
	flag.StringVar(&bucketName, "bucket", "", "The name of the bucket.")
	flag.StringVar(&objectName, "object", "", "The name of the object.")
}

func main() {
	flag.Parse()

	var position = int64(0) // First append always starts at position 0.

	if len(bucketName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, bucket name required")
	}

	if len(region) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, region required")
	}

	if len(objectName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, object name required")
	}

	cfg := oss.LoadDefaultConfig().
		WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
		WithRegion(region)

	client := oss.NewClient(cfg)

	// First append: position starts at 0.
	request := &oss.AppendObjectRequest{
		Bucket:   oss.Ptr(bucketName),
		Key:      oss.Ptr(objectName),
		Position: oss.Ptr(position),
		Body:     strings.NewReader("hi append object"),
	}

	result, err := client.AppendObject(context.TODO(), request)
	if err != nil {
		log.Fatalf("failed to append object %v", err)
	}

	// Second append: use NextPosition from the previous result.
	request = &oss.AppendObjectRequest{
		Bucket:   oss.Ptr(bucketName),
		Key:      oss.Ptr(objectName),
		Position: oss.Ptr(result.NextPosition),
		Body:     strings.NewReader("hi append object"),
	}

	result, err = client.AppendObject(context.TODO(), request)
	if err != nil {
		log.Fatalf("failed to append object %v", err)
	}

	log.Printf("append object result:%#v\n", result)
}

Common scenarios

Display a progress bar during append upload

Set the ProgressFn field on AppendObjectRequest to receive progress callbacks. The callback receives three values: bytes added in this call (increment), total bytes transferred so far (transferred), and total object size (total).

package main

import (
	"context"
	"flag"
	"fmt"
	"log"
	"strings"

	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
)

var (
	region     string
	bucketName string
	objectName string
)

func init() {
	flag.StringVar(&region, "region", "", "The region in which the bucket is located.")
	flag.StringVar(&bucketName, "bucket", "", "The name of the bucket.")
	flag.StringVar(&objectName, "object", "", "The name of the object.")
}

func main() {
	flag.Parse()

	var position = int64(0)

	if len(bucketName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, bucket name required")
	}

	if len(region) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, region required")
	}

	if len(objectName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, object name required")
	}

	cfg := oss.LoadDefaultConfig().
		WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
		WithRegion(region)

	client := oss.NewClient(cfg)

	// ProgressFn is called periodically to report upload progress.
	progressFn := func(increment, transferred, total int64) {
		fmt.Printf("increment:%v, transferred:%v, total:%v\n", increment, transferred, total)
	}

	// First append with progress tracking.
	request := &oss.AppendObjectRequest{
		Bucket:     oss.Ptr(bucketName),
		Key:        oss.Ptr(objectName),
		Position:   oss.Ptr(position),
		Body:       strings.NewReader("hi append object"),
		ProgressFn: progressFn,
	}

	result, err := client.AppendObject(context.TODO(), request)
	if err != nil {
		log.Fatalf("failed to append object %v", err)
	}

	// Second append: use NextPosition and continue tracking progress.
	request = &oss.AppendObjectRequest{
		Bucket:     oss.Ptr(bucketName),
		Key:        oss.Ptr(objectName),
		Position:   oss.Ptr(result.NextPosition),
		Body:       strings.NewReader("hi append object"),
		ProgressFn: progressFn,
	}

	result, err = client.AppendObject(context.TODO(), request)
	if err != nil {
		log.Fatalf("failed to append object %v", err)
	}

	log.Printf("append object result:%#v\n", *result.VersionId)
}

What's next

For complete sample code, see GitHub examples and the Developer Guide.
For the full AppendFile API reference, see AppendFile.
For the full AppendObject API reference, see AppendObject.