Object Storage Service:Multipart upload

Usage notes

• In this topic, the public endpoint of the China (Hangzhou) region is used. If you want to access OSS by using other Alibaba Cloud services in the same region as OSS, use an internal endpoint. For more information about the regions and endpoints supported by OSS, see Regions and endpoints.

• In this topic, access credentials are obtained from environment variables. For more information about how to configure access credentials, see Configure access credentials.

• In this topic, an OSSClient instance is created by using an OSS endpoint. If you want to create an OSSClient instance by using custom domain names or Security Token Service (STS), see Initialization.

• To perform multipart upload, you must have the oss:PutObject permission. For more information, see Attach a custom policy to a RAM user.

Process

To upload an object by using multipart upload, perform the following steps:

1. Initiate a multipart upload task.

   Call the Bucket.InitiateMultipartUpload method to obtain a unique upload ID in OSS.

2. Upload parts.

   Call the Bucket.UploadPart method to upload the parts.

   Note

   • For parts that are uploaded by a multipart upload task with a specific upload ID, part numbers identify their relative positions in an object. If you upload a part and reuse its part number to upload another part, the new part overwrites the original part.

   • OSS includes the MD5 hash of each uploaded part in the ETag header in the response.

   • OSS calculates the MD5 hash of uploaded parts and compares the MD5 hash with the MD5 hash that is calculated by OSS SDK for Go. If the two hashes are different, OSS returns the InvalidDigest error code.

3. Complete the multipart upload task.

   After you upload all parts, call the Bucket.CompleteMultipartUpload method to combine the parts into a complete object.

Sample code of multipart upload

The following code provides an example on how to perform multipart upload:

package main

import (
    "fmt"
    "os"
    "time"

    "github.com/aliyun/aliyun-oss-go-sdk/oss"
)

func main() {
	// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
	provider, err := oss.NewEnvironmentVariableCredentialsProvider()
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// Create an OSSClient instance. 
	// Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. Specify your actual endpoint. 
	client, err := oss.New("yourEndpoint", "", "", oss.SetCredentialsProvider(&provider))
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
    // Specify the name of the bucket. 
    bucketName := "examplebucket"
    // Specify the full path of the object. Do not include the bucket name in the full path. 
    objectName := "exampleobject.txt"
    // Specify the full path of the local file that you want to upload. By default, if you do not specify the full path of the local file, the local file is uploaded from the path of the project to which the sample program belongs. 
    locaFilename := "D:\\localpath\\examplefile.txt"
   
    bucket, err := client.Bucket(bucketName)
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
    // Split the local file into three parts. 
    chunks, err := oss.SplitFileByPartNum(locaFilename, 3)
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
    fd, err := os.Open(locaFilename)
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
    defer fd.Close()

    // Specify the validity period of the request. 
    expires := time.Date(2049, time.January, 10, 23, 0, 0, 0, time.UTC)
    // The following code provides an example on how to specify the request headers when you initiate a multipart upload task. 
    options := []oss.Option{
        oss.MetadataDirective(oss.MetaReplace),
        oss.Expires(expires),
        // Specify the caching behavior of the web page when the object is downloaded. 
        // oss.CacheControl("no-cache"),
        // Specify the name of the object when it is downloaded. 
        // oss.ContentDisposition("attachment;filename=FileName.txt"),
        // Specify the content encoding format of the object. 
        // oss.ContentEncoding("gzip"),
        // Specify the method that is used to encode the object name in the response. Only URL encoding is supported. 
        // oss.EncodingType("url"),
        // Specify the storage class of the object. 
        // oss.ObjectStorageClass(oss.StorageStandard),
    }

    // Step 1: Initiate a multipart upload task. 
    imur, err := bucket.InitiateMultipartUpload(objectName, options...)
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
    // Step 2: Upload the parts. 
    var parts []oss.UploadPart
    for _, chunk := range chunks {
        fd.Seek(chunk.Offset, os.SEEK_SET)
        // Call the UploadPart method to upload each part. 
        part, err := bucket.UploadPart(imur, fd, chunk.Size, chunk.Number)
        if err != nil {
            fmt.Println("Error:", err)
            os.Exit(-1)
        }
        parts = append(parts, part)
    }

    // Set the access control list (ACL) of the object to private. By default, the object inherits the ACL of the bucket. 
    objectAcl := oss.ObjectACL(oss.ACLPrivate)

    // Step 3: Complete the multipart upload task. 
    cmur, err := bucket.CompleteMultipartUpload(imur, parts, objectAcl)
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
    fmt.Println("cmur:", cmur)
}

Cancel a multipart upload task

You can call the bucket.AbortMultipartUpload method to cancel a multipart upload task. If you cancel a multipart upload task, you cannot use the upload ID to manage parts. The uploaded parts are deleted.

The following sample code provides an example on how to cancel a multipart upload task:

package main

import (
	"fmt"
	"github.com/aliyun/aliyun-oss-go-sdk/oss"
	"os"
)

func main() {
	// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
	provider, err := oss.NewEnvironmentVariableCredentialsProvider()
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// Create an OSSClient instance. 
	// Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. Specify your actual endpoint. 
	client, err := oss.New("yourEndpoint", "", "", oss.SetCredentialsProvider(&provider))
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// Specify the name of the bucket. 
	bucket, err := client.Bucket("examplebucket")
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// Initiate a multipart upload task. 
	// Specify the full path of the object. Do not include the bucket name in the full path. 
	imur, err := bucket.InitiateMultipartUpload("exampleobject.txt")
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// Cancel the multipart upload task. 
	err = bucket.AbortMultipartUpload(imur)
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
}

List uploaded parts

The following code provides an example on how to list the parts that are uploaded in a specific multipart upload task:

package main

import (
	"fmt"
	"github.com/aliyun/aliyun-oss-go-sdk/oss"
	"os"
)

func main() {
	// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
	provider, err := oss.NewEnvironmentVariableCredentialsProvider()
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// Create an OSSClient instance. 
	// Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. Specify your actual endpoint. 
	client, err := oss.New("yourEndpoint", "", "", oss.SetCredentialsProvider(&provider))
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// Specify the name of the bucket. 
	bucketName := "examplebucket"
	// Specify the full path of the object. Do not include the bucket name in the full path. 
	objectName := "exampleobject.txt"
	// Specify the full path of the local file that you want to upload. By default, if you do not specify the full path of the local file, the local file is uploaded from the path of the project to which the sample program belongs. 
	locaFilename := "D:\\localpath\\examplefile.txt"

	bucket, err := client.Bucket(bucketName)
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// Split the local file into three parts. 
	chunks, err := oss.SplitFileByPartNum(locaFilename, 3)
	fd, err := os.Open(locaFilename)
	defer fd.Close()
	// Initiate a multipart upload task. 
	imur, err := bucket.InitiateMultipartUpload(objectName)
	uploadID := imur.UploadID
	fmt.Println("InitiateMultipartUpload UploadID: ", uploadID)
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// Upload the parts. 
	var parts []oss.UploadPart
	for _, chunk := range chunks {
		fd.Seek(chunk.Offset, os.SEEK_SET)
		// Call the UploadPart method to upload each part. 
		part, err := bucket.UploadPart(imur, fd, chunk.Size, chunk.Number)
		if err != nil {
			fmt.Println("Error:", err)
			os.Exit(-1)
		}
		fmt.Println("UploadPartNumber: ", part.PartNumber, ", ETag: ", part.ETag)
		parts = append(parts, part)
	}
	// Call the InitiateMultipartUploadResult method to list the uploaded parts. 
	lsRes, err := bucket.ListUploadedParts(imur)
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// Display the uploaded parts. 
	fmt.Println("\nParts:", lsRes.UploadedParts)
	for _, upload := range lsRes.UploadedParts {
		fmt.Println("List PartNumber:  ", upload.PartNumber, ", ETag: ", upload.ETag, ", LastModified: ", upload.LastModified)
	}
	// List all uploaded parts based on the result that is obtained by using the InitiateMultipartUploadResult operation. The result is generated based on the specified object name and upload ID. 
	var imur_with_uploadid oss.InitiateMultipartUploadResult
	imur_with_uploadid.Key = objectName
	imur_with_uploadid.UploadID = uploadID
	// List uploaded parts. 
	lsRes, err = bucket.ListUploadedParts(imur_with_uploadid)
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// Display the uploaded parts. 
	fmt.Println("\nListUploadedParts by UploadID: ", uploadID)
	for _, upload := range lsRes.UploadedParts {
		fmt.Println("List PartNumber:  ", upload.PartNumber, ", ETag: ", upload.ETag, ", LastModified: ", upload.LastModified)
	}
	// Complete the multipart upload task. 
	cmur, err := bucket.CompleteMultipartUpload(imur, parts)
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	fmt.Println("cmur:", cmur)
}

List multipart upload tasks

You can call the Bucket.ListMultipartUploads method to list all ongoing multipart upload tasks those are tasks that have been initiated but are not completed or canceled. The following table describes the parameters that you can configure to list multipart upload tasks.

• List multipart upload tasks by using default values of the parameters

  package main
  
  import (
  	"fmt"
  	"github.com/aliyun/aliyun-oss-go-sdk/oss"
  	"os"
  )
  
  func main() {
  	// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
  	provider, err := oss.NewEnvironmentVariableCredentialsProvider()
  	if err != nil {
  		fmt.Println("Error:", err)
  		os.Exit(-1)
  	}
  	// Create an OSSClient instance. 
  	// Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. Specify your actual endpoint. 
  	client, err := oss.New("yourEndpoint", "", "", oss.SetCredentialsProvider(&provider))
  	if err != nil {
  		fmt.Println("Error:", err)
  		// Specify the name of the bucket. 
  		bucketName := "examplebucket"
  		bucket, err := client.Bucket(bucketName)
  		if err != nil {
  			fmt.Println("Error:", err)
  			os.Exit(-1)
  		}
  		// List all multipart upload tasks. 
  		keyMarker := ""
  		uploadIdMarker := ""
  		for {
  			// By default, a maximum of 1,000 records can be returned at a time. 
  			lsRes, err := bucket.ListMultipartUploads(oss.KeyMarker(keyMarker), oss.UploadIDMarker(uploadIdMarker))
  			if err != nil {
  				fmt.Println("Error:", err)
  				os.Exit(-1)
  			}
  			// Display the multipart upload tasks. 
  			for _, upload := range lsRes.Uploads {
  				fmt.Println("Upload: ", upload.Key, ", UploadID: ", upload.UploadID)
  			}
  			if lsRes.IsTruncated {
  				keyMarker = lsRes.NextKeyMarker
  				uploadIdMarker = lsRes.NextUploadIDMarker
  			} else {
  				break
  			}
  		}
  	}
  }
  

• Set the Prefix parameter to file and list multipart upload tasks

      lsRes, err := bucket.ListMultipartUploads(oss.Prefix("file"))
      if err != nil {
          fmt.Println("Error:", err)
          os.Exit(-1)
      }
      fmt.Println("Uploads:", lsRes.Uploads)

• Specify that a maximum of 100 multipart upload tasks are returned

      lsRes, err := bucket.ListMultipartUploads(oss.MaxUploads(100))
      if err != nil {
          fmt.Println("Error:", err)
          os.Exit(-1)
      }
      fmt.Println("Uploads:", lsRes.Uploads)

• Set the Prefix parameter to file and specify that a maximum of 100 multipart upload tasks are returned

      lsRes, err := bucket.ListMultipartUploads(oss.Prefix("file"), oss.MaxUploads(100))
      if err != nil {
          fmt.Println("Error:", err)
          os.Exit(-1)
      }
      fmt.Println("Uploads:", lsRes.Uploads)

References

• For the complete sample code that is used to perform multipart upload, visit GitHub.

• A multipart upload involves three API operations. For more information about the operations, see the following topics:

  • InitiateMultipartUpload

  • UploadPart

  • CompleteMultipartUpload

• For more information about the API operation that you can call to cancel a multipart upload task, see AbortMultipartUpload.

• For more information about the API operation that you can call to list uploaded parts, see ListParts.

• For more information about the API operation that you can call to list all ongoing multipart upload tasks, see ListMultipartUploads. Ongoing multipart upload tasks are tasks that have been initiated but are not completed or canceled.