Object Storage Service:Multipart copy using OSS SDK for Go 2.0

Usage notes

• The sample code uses the region ID cn-hangzhou of the China (Hangzhou) region and the public endpoint by default. To access resources by using other Alibaba Cloud services in the same region as the bucket, use the internal endpoint. For more information about OSS regions and endpoints, see Regions and Endpoints.

• Access credentials are obtained from environment variables. For more information about how to configure access credentials, see Configure access credentials.

• To copy an object, you must have the read permissions on the source object and the read and write permissions on the destination bucket.

• The source bucket and destination bucket must be located in the same region. For example, objects in a bucket located in the China (Hangzhou) region cannot be copied to another bucket located in the China (Qingdao) region.

• Make sure that no retention policies are configured for the source bucket and the destination bucket. Otherwise, the following error message is returned: The object you specified is immutable.

Method

func (c *Client) UploadPartCopy(ctx context.Context, request *UploadPartCopyRequest, optFns ...func(*Options)) (*UploadPartCopyResult, error)

Request parameters

Response parameters

Multipart copy process

To copy an object by using multipart copy, perform the following steps:

1. Initiate a multipart upload task.

   Use the Client.InitiateMultipartUpload method to obtain a unique upload ID in OSS.

2. Upload parts.

   Use the Client.UploadPartCopy method to upload the parts.

   Note

   • For parts that are uploaded by running a multipart upload task with a specific upload ID, the part numbers identify their relative positions in an object. If you upload a part and reuse its part number to upload another part, the new part overwrites the original part.

   • OSS includes the MD5 hash of each uploaded part in the ETag header in the response.

   • OSS calculates the MD5 hash of uploaded parts and compares the MD5 hash with the MD5 hash that is calculated by OSS SDK for Go. If the two hashes are different, OSS returns the InvalidDigest error code.

3. Complete the multipart upload task.

   After all parts are uploaded, use the Client.CompleteMultipartUpload method to combine these parts into a complete object.

Examples

The following sample code shows how to copy an object from a source bucket to a destination bucket by using multipart copy and combine the parts into a complete object:

package main

import (
	"context"
	"flag"
	"log"
	"sync"

	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
	"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
)

// Specify the global variables.
var (
	region         string // The region.
	srcBucketName string // The name of the source bucket.
	srcObjectName string // The name of the source object.
	destBucketName string // The name of the destination bucket.
	destObjectName string // The name of the destination object.
)

// Specify the init function used to initialize command line parameters.
func init() {
	flag.StringVar(&region, "region", "", "The region in which the bucket is located.")
	flag.StringVar(&srcBucketName, "src-bucket", "", "The name of the source bucket.")
	flag.StringVar(&srcObjectName, "src-object", "", "The name of the source object.")
	flag.StringVar(&destBucketName, "dest-bucket", "", "The name of the destination bucket.")
	flag.StringVar(&destObjectName, "dest-object", "", "The name of the destination object.")
}

func main() {
	// Parse command line parameters.
	flag.Parse()

	// Specify the upload ID.
	var uploadId string

	// Check whether the source bucket name is empty.
	if len(srcBucketName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, source bucket name required")
	}

	// Check whether the region is empty.
	if len(region) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, region required")
	}

	// If the destination bucket name is not specified, the source bucket name is used.
	if len(destBucketName) == 0 {
		destBucketName = srcBucketName
	}

	// Check whether the source object name is empty.
	if len(srcObjectName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, source object name required")
	}

	// Check whether the destination object name is empty.
	if len(destObjectName) == 0 {
		flag.PrintDefaults()
		log.Fatalf("invalid parameters, destination object name required")
	}

	// Load the default configurations and specify the credential provider and region.
	cfg := oss.LoadDefaultConfig().
		WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
		WithRegion(region)

	// Create an OSS client.
	client := oss.NewClient(cfg)

	// Initialize the multipart upload request.
	initRequest := &oss.InitiateMultipartUploadRequest{
		Bucket: oss.Ptr(destBucketName),
		Key:    oss.Ptr(destObjectName),
	}
	initResult, err := client.InitiateMultipartUpload(context.TODO(), initRequest)
	if err != nil {
		log.Fatalf("failed to initiate multipart upload %v", err)
	}

	// Display the result of multipart copy.
	log.Printf("initiate multipart upload result:%#v\n", *initResult.UploadId)
	uploadId = *initResult.UploadId

	// Initialize the wait group and mutual exclusion (mutex).
	var wg sync.WaitGroup
	var parts []oss.UploadPart
	count := 3
	var mu sync.Mutex

	// Start multiple goroutines for multipart upload.
	for i := 0; i < count; i++ {
		wg.Add(1)
		go func(partNumber int, i int) {
			defer wg.Done()

			// Create the multipart upload request.
			partRequest := &oss.UploadPartCopyRequest{
				Bucket:       oss.Ptr(destBucketName), // The name of the destination bucket.
				Key:          oss.Ptr(destObjectName), // The name of the destination object.
				SourceBucket: oss.Ptr(srcBucketName), // The name of the source bucket.
				SourceKey:     oss.Ptr(srcObjectName), // The name of the source object.
				PartNumber: int32(partNumber), // The part number.
				const uploadId = '<Upload Id>'; // The upload ID of the multipart upload task.
			}

			// Send the multipart upload request.
			partResult, err := client.UploadPartCopy(context.TODO(), partRequest)
			if err != nil {
				log.Fatalf("failed to upload part copy %d: %v", partNumber, err)
			}

			// Record the multipart upload result.
			part := oss.UploadPart{
				PartNumber: partRequest.PartNumber,
				ETag:       partResult.ETag,
			}

			// Use the mutex to protect shared data.
			mu.Lock()
			parts = append(parts, part)
			mu.Unlock()
		}(i+1, i)
	}

	// Wait until all goroutines are complete.
	wg.Wait()

	// Complete the multipart upload request.
	request := &oss.CompleteMultipartUploadRequest{
		Bucket:   oss.Ptr(destBucketName),
		Key:      oss.Ptr(destObjectName),
		UploadId: oss.Ptr(uploadId),
		CompleteMultipartUpload: &oss.CompleteMultipartUpload{
			Parts: parts,
		},
	}
	result, err := client.CompleteMultipartUpload(context.TODO(), request)
	if err != nil {
		log.Fatalf("failed to complete multipart upload %v", err)
	}

	// Display the result.
	log.Printf("complete multipart upload result:%#v\n", result)
}

References

• For the complete sample code for multipart copy, visit GitHub.

• For more information about the InitiateMultipartUpload operation, visit InitiateMultipartUpload.

• For more information about the UploadPartCopy operation, visit UploadPartCopy.

• For more information about the CompleteMultipartUpload operation, visit CompleteMultipartUpload.