Object Storage Service:Multipart upload

Usage notes

• In this topic, the public endpoint of the China (Hangzhou) region is used. If you want to access OSS by using other Alibaba Cloud services in the same region as OSS, use an internal endpoint. For more information about the regions and endpoints supported by OSS, see Regions and endpoints.
• In this topic, an OSSClient instance is created by using an OSS endpoint. If you want to create an OSSClient by using custom domain names or STS, see Initialization.
• The oss:PutObject permission is required to perform multipart upload. For more information, see Attach a custom policy to a RAM user.

Multipart upload process

To implement multipart upload, perform the following operations:

1. Initiate a multipart upload task.

   Call the Bucket.InitiateMultipartUpload method to obtain a unique upload ID in OSS.

2. Upload parts.

   Call the Bucket.UploadPart method to upload the parts.

   Note

   • For parts that are uploaded by a multipart upload task with a specific upload ID, part numbers identify their relative positions in an object. If you upload a part and use its part number again to upload another part, the latter part overwrites the former part.
   • OSS includes the MD5 hash of part data in the ETag header and returns the MD5 hash to the user.
   • OSS calculates the MD5 hash of uploaded data and compares the MD5 hash with the MD5 hash calculated by the SDK. If the two hashes are different, the InvalidDigest error code is returned.
3. Complete the multipart upload task.

   After all parts are uploaded, call the Bucket.CompleteMultipartUpload method to combine the parts into a complete object.

Complete sample code

The following code provides an example on how to perform multipart upload:

package main

import (
    "fmt"
    "os"

    "github.com/aliyun/aliyun-oss-go-sdk/oss"
)

func main() {
    // Create an OSSClient instance. 
    // Set yourEndpoint to the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set yourEndpoint to https://oss-cn-hangzhou.aliyuncs.com. 
    // Security risks may arise if you use the AccessKey pair of an Alibaba Cloud account to access OSS because the account has permissions on all API operations. We recommend that you use a RAM user to call API operations or perform routine operations and maintenance. To create a RAM user, log on to the RAM console. 
    client, err := oss.New("yourEndpoint", "yourAccessKeyId", "yourAccessKeySecret")
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
    // Specify the name of the bucket. 
    bucketName := "examplebucket"
    // Specify the full path of the object. The full path of the object cannot contain the bucket name. 
    objectName := "exampleobject.txt"
    // Specify the full path of the local file that you want to upload. By default, if you do not specify the full path of the local file, the local file is uploaded from the path of the project to which the sample program belongs. 
    locaFilename := "D:\\localpath\\examplefile.txt"

    bucket, err := client.Bucket(bucketName)
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
    // Split the local file into three parts. 
    chunks, err := oss.SplitFileByPartNum(locaFilename, 3)
    fd, err := os.Open(locaFilename)
    defer fd.Close()
    
    // Specify the expiration time. 
    expires := time.Date(2049, time.January, 10, 23, 0, 0, 0, time.UTC)
    // The following code provides an example on how to specify the request headers when you initiate a multipart upload task. 
    options := []oss.Option{
        oss.MetadataDirective(oss.MetaReplace),
        oss.Expires(expires),
        // Specify the caching behavior of the web page when the object is downloaded. 
        // oss.CacheControl("no-cache"),
        // Specify the name of the object when the object is downloaded. 
        // oss.ContentDisposition("attachment;filename=FileName.txt"),
        // Specify the encoding format for the content of the object. 
        // oss.ContentEncoding("gzip"),
        // Specify the method that is used to encode the object name in the response. Only URL encoding is supported. 
        // oss.EncodingType("url"),
        // Specify the storage class of the object. 
        // oss.ObjectStorageClass(oss.StorageStandard),
    }

    // Step 1: Initiate a multipart upload task and set the storage class to Standard. 
    imur, err := bucket.InitiateMultipartUpload(objectName, options...)
    // Step 2: Upload parts. 
    var parts []oss.UploadPart
    for _, chunk := range chunks {
        fd.Seek(chunk.Offset, os.SEEK_SET)
        // Call the UploadPart method to upload each part. 
        part, err := bucket.UploadPart(imur, fd, chunk.Size, chunk.Number)
        if err != nil {
            fmt.Println("Error:", err)
            os.Exit(-1)
        }
        parts = append(parts, part)
    }

    // Set the access control list (ACL) of the object to public-read. By default, the object inherits the ACL of the bucket. 
    objectAcl := oss.ObjectACL(oss.ACLPublicRead)

    // Step 3: Complete the multipart upload task and set the ACL of the object to public-read. 
    cmur, err := bucket.CompleteMultipartUpload(imur, parts, objectAcl)
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
    fmt.Println("cmur:", cmur)
}

Cancel a multipart upload task

You can call the bucket.AbortMultipartUpload method to cancel a multipart upload task. If you cancel a multipart upload task, you cannot use the upload ID to manage parts. The uploaded parts are deleted.

The following code provides an example on how to cancel a multipart upload task:

package main
import (
    "fmt"
    "os"
    "github.com/aliyun/aliyun-oss-go-sdk/oss"
)
func main() {
    // Create an OSSClient instance. 
    // Set yourEndpoint to the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set yourEndpoint to https://oss-cn-hangzhou.aliyuncs.com. 
    // Security risks may arise if you use the AccessKey pair of an Alibaba Cloud account to access OSS because the account has permissions on all API operations. We recommend that you use a RAM user to call API operations or perform routine operations and maintenance. To create a RAM user, log on to the RAM console. 
    client, err := oss.New("yourEndpoint", "yourAccessKeyId", "yourAccessKeySecret")

    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
    // Specify the bucket name. 
    bucket, err := client.Bucket("examplebucket")
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
    // Initiate a multipart upload task. 
    // Specify the full path of the object. The full path of the object cannot contain the bucket name. 
    imur, err := bucket.InitiateMultipartUpload("exampleobject.txt")
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
    // Cancel the multipart upload task. 
    err = bucket.AbortMultipartUpload(imur)
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
}

List the uploaded parts

The following code provides an example on how to list the parts that are uploaded in the specified multipart upload task:

package main
import (
    "fmt"
    "os"
    "github.com/aliyun/aliyun-oss-go-sdk/oss"
)
func main() {
    // Create an OSSClient instance. 
    // Set yourEndpoint to the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set yourEndpoint to https://oss-cn-hangzhou.aliyuncs.com. 
    // Security risks may arise if you use the AccessKey pair of an Alibaba Cloud account to access OSS because the account has permissions on all API operations. We recommend that you use a RAM user to call API operations or perform routine operations and maintenance. To create a RAM user, log on to the RAM console. 
    client, err := oss.New("yourEndpoint", "yourAccessKeyId", "yourAccessKeySecret")
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
     // Specify the bucket name. 
    bucketName := "examplebucket"
    // Specify the full path of the object. The full path of the object cannot contain the bucket name. 
    objectName := "exampleobject.txt"
    // Specify the full path of the local file that you want to upload. By default, if you do not specify the local file, the local file is uploaded from the path of the project to which the sample program belongs. 
    locaFilename := "D:\\localpath\\examplefile.txt"
  

    bucket, err := client.Bucket(bucketName)
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
    // Split the local file into three parts. 
    chunks, err := oss.SplitFileByPartNum(locaFilename, 3)
    fd, err := os.Open(locaFilename)
    defer fd.Close()
    // Initiate a multipart upload task. 
    imur, err := bucket.InitiateMultipartUpload(objectName)
    uploadID := imur.UploadID
    fmt.Println("InitiateMultipartUpload UploadID: ", uploadID)
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
    // Upload parts. 
    var parts []oss.UploadPart
    for _, chunk := range chunks {
        fd.Seek(chunk.Offset, os.SEEK_SET)
        // Call the UploadPart method to upload each part. 
        part, err := bucket.UploadPart(imur, fd, chunk.Size, chunk.Number)
        if err != nil {
            fmt.Println("Error:", err)
            os.Exit(-1)
        }
        fmt.Println("UploadPartNumber: ", part.PartNumber, ", ETag: ", part.ETag)
        parts = append(parts, part)
    }
    // List the uploaded parts based on InitiateMultipartUploadResult. 
    lsRes, err := bucket.ListUploadedParts(imur)
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
    // Display the uploaded parts.  
    fmt.Println("\nParts:", lsRes.UploadedParts)
    for _, upload := range lsRes.UploadedParts {
        fmt.Println("List PartNumber:  ", upload.PartNumber, ", ETag: " ,upload.ETag, ", LastModified: ", upload.LastModified)
    }
    // Specify objectName and uploadID to obtain InitiateMultipartUploadResult. List all uploaded parts based on the result. 
    var imur_with_uploadid oss.InitiateMultipartUploadResult
    imur_with_uploadid.Key = objectName
    imur_with_uploadid.UploadID = uploadID
    // List the uploaded parts. 
    lsRes, err = bucket.ListUploadedParts(imur_with_uploadid)
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
    // Display the uploaded parts. 
    fmt.Println("\nListUploadedParts by UploadID: ", uploadID)
    for _, upload := range lsRes.UploadedParts {
        fmt.Println("List PartNumber:  ", upload.PartNumber, ", ETag: " ,upload.ETag, ", LastModified: ", upload.LastModified)
    }
    // Complete the multipart upload task. 
    cmur, err := bucket.CompleteMultipartUpload(imur, parts)
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
    fmt.Println("cmur:", cmur)
}

List multipart upload tasks

You can call the Bucket.ListMultipartUploads method to list all ongoing multipart upload tasks, which include tasks that have been initiated but are not completed and tasks that are canceled. The following table describes the parameters that you can configure to list these tasks.

• List multipart upload tasks by using default parameter values

  package main
  import (
      "fmt"
      "os"
      "github.com/aliyun/aliyun-oss-go-sdk/oss"
  )
  func main() {
      // Create an OSSClient instance. 
      // Set yourEndpoint to the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set yourEndpoint to https://oss-cn-hangzhou.aliyuncs.com. 
      // Security risks may arise if you use the AccessKey pair of an Alibaba Cloud account to access OSS because the account has permissions on all API operations. We recommend that you use a RAM user to call API operations or perform routine operations and maintenance. To create a RAM user, log on to the RAM console. 
      client, err := oss.New("yourEndpoint", "yourAccessKeyId", "yourAccessKeySecret")
      if err != nil {
          fmt.Println("Error:", err)
          os.Exit(-1)
      }
      // Specify the bucket name. 
      bucketName := "examplebucket"
      bucket, err := client.Bucket(bucketName)
      if err != nil {
          fmt.Println("Error:", err)
          os.Exit(-1)
      }
      // List all multipart upload tasks. 
      keyMarker := ""
      uploadIdMarker := ""
      for {
          // By default, a maximum of 1,000 records can be returned at a time.  
          lsRes, err := bucket.ListMultipartUploads(oss.KeyMarker(keyMarker), oss.UploadIDMarker(uploadIdMarker))
          if err != nil {
              fmt.Println("Error:", err)
              os.Exit(-1)
          }
          // Display the multipart upload tasks.  
          for _, upload := range lsRes.Uploads {
              fmt.Println("Upload: ", upload.Key, ", UploadID: ",upload.UploadID)
          }
          if lsRes.IsTruncated {
              keyMarker = lsRes.NextKeyMarker
              uploadIdMarker = lsRes.NextUploadIDMarker
          } else {
              break
          }
      }
  }

• Specify file as the prefix

      lsRes, err := bucket.ListMultipartUploads(oss.Prefix("file"))
      if err != nil {
          fmt.Println("Error:", err)
          os.Exit(-1)
      }
      fmt.Println("Uploads:", lsRes.Uploads)

• Specify that a maximum of 100 multipart upload tasks are returned

      lsRes, err := bucket.ListMultipartUploads(oss.MaxUploads(100))
      if err != nil {
          fmt.Println("Error:", err)
          os.Exit(-1)
      }
      fmt.Println("Uploads:", lsRes.Uploads)

• Specify file as the prefix and specify that a maximum of 100 multipart upload tasks are returned

      lsRes, err := bucket.ListMultipartUploads(oss.Prefix("file"), oss.MaxUploads(100))
      if err != nil {
          fmt.Println("Error:", err)
          os.Exit(-1)
      }
      fmt.Println("Uploads:", lsRes.Uploads)

References

• For more information about the complete sample code of multipart upload, visit GitHub.
• The following API operations are required to perform multipart upload:
  • The API operation that you can call to initiate a multipart upload task. For more information, see InitiateMultipartUpload.
  • The API operation that you can call to upload data by part. For more information, see UploadPart.
  • The API operation that you can call to complete a multipart upload task. For more information, see CompleteMultipartUpload.
• For more information about the API operation that you can call to cancel a multipart upload task, see AbortMultipartUpload.
• For more information about the API operation that you can call to list the uploaded parts, see ListParts.
• For more information about the API operation that you can call to list all ongoing multipart upload tasks, see ListMultipartUploads. Ongoing multipart upload tasks include tasks that have been initiated but are not completed and tasks that are canceled.