Object Storage Service:Multipart upload

Usage notes

• In this topic, the public endpoint of the China (Hangzhou) region is used. If you want to access OSS by using other Alibaba Cloud services in the same region as OSS, use an internal endpoint For more information about the regions and endpoints supported by OSS, see Regions and endpoints.
• In this topic, an OSSClient instance is created by using an OSS endpoint. If you want to create an OSSClient instance by using custom domain names or STS, see Initialization.
• The oss:PutObject permission is required to perform multipart upload. For more information, see Attach a custom policy to a RAM user.

Procedure

To implement multipart upload, perform the following steps:

1. Initiate a multipart upload task.

   Call the Bucket.InitiateMultipartUpload method to obtain a unique upload ID in OSS.

2. Upload parts.

   Call the Bucket.UploadPart method to upload the parts.

   Note

   • For parts that are uploaded by a multipart upload task with a specific upload ID, part numbers identify their relative positions in an object. If you upload a part and reuse its part number to upload another part, the new part overwrites the original part.
   • OSS includes the MD5 hash of each uploaded part in the ETag header in the response.
   • OSS calculates the MD5 hash of uploaded parts and compares the MD5 hash with the MD5 hash that is calculated by OSS SDK for Go. If the two hashes are different, OSS returns the InvalidDigest error code.
3. Complete the multipart upload task.

   After you upload all parts, call the Bucket.CompleteMultipartUpload method to combine the parts into a complete object.

Examples

The following code provides an example on how to perform multipart upload:

package main

import (
    "fmt"
    "os"
    "time"

    "github.com/aliyun/aliyun-oss-go-sdk/oss"
)

func main() {
    // Create an OSSClient instance. 
    // Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. 
    // The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. Using these credentials to perform operations in OSS is a high-risk operation. We recommend that you use a RAM user to call API operations or perform routine O&M. To create a RAM user, log on to the RAM console. 
    client, err := oss.New("yourEndpoint", "yourAccessKeyId", "yourAccessKeySecret")
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
    // Specify the name of the bucket. 
    bucketName := "examplebucket"
    // Specify the full path of the object. Do not include the bucket name in the full path of the object. 
    objectName := "exampleobject.txt"
    // Specify the full path of the local file. By default, if you do not specify the full path of the local file, the local file is uploaded from the path of the project to which the sample program belongs. 
    locaFilename := "D:\\localpath\\examplefile.txt"
   
    bucket, err := client.Bucket(bucketName)
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
    // Split the local file into three parts. 
    chunks, err := oss.SplitFileByPartNum(locaFilename, 3)
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
    fd, err := os.Open(locaFilename)
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
    defer fd.Close()

    // Specify the validity period of the request. 
    expires := time.Date(2049, time.January, 10, 23, 0, 0, 0, time.UTC)
    // The following code provides an example on how to specify the request headers when you initiate a multipart upload task. 
    options := []oss.Option{
        oss.MetadataDirective(oss.MetaReplace),
        oss.Expires(expires),
        // Specify the caching behavior of the web page when the object is downloaded. 
        // oss.CacheControl("no-cache"),
        // Specify the name of the downloaded object. 
        // oss.ContentDisposition("attachment;filename=FileName.txt"),
        // Specify the content encoding format of the object. 
        // oss.ContentEncoding("gzip"),
        // Specify the method that is used to encode the object name in the response. Only URL encoding is supported. 
        // oss.EncodingType("url"),
        // Specify the storage class of the object. 
        // oss.ObjectStorageClass(oss.StorageStandard),
    }

    // Step 1: Initiate a multipart upload task. 
    imur, err := bucket.InitiateMultipartUpload(objectName, options...)
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
    // Step 2: Upload the parts. 
    var parts []oss.UploadPart
    for _, chunk := range chunks {
        fd.Seek(chunk.Offset, os.SEEK_SET)
        // Call the UploadPart method to upload each part. 
        part, err := bucket.UploadPart(imur, fd, chunk.Size, chunk.Number)
        if err != nil {
            fmt.Println("Error:", err)
            os.Exit(-1)
        }
        parts = append(parts, part)
    }

    // Set the access control list (ACL) of the object to private. By default, the object inherits the ACL of the bucket. 
    objectAcl := oss.ObjectACL(oss.ACLPrivate)

    // Step 3: Complete multipart upload. 
    cmur, err := bucket.CompleteMultipartUpload(imur, parts, objectAcl)
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
    fmt.Println("cmur:", cmur)
}

Cancel a multipart upload task

You can call the bucket.AbortMultipartUpload method to cancel a multipart upload task. If you cancel a multipart upload task, you cannot use the upload ID to manage parts. The uploaded parts are deleted.

The following sample code provides an example on how to cancel a multipart upload task:

package main
import (
    "fmt"
    "os"
    "github.com/aliyun/aliyun-oss-go-sdk/oss"
)
func main() {
    // Create an OSSClient instance. 
    // Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. 
    // The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. Using these credentials to perform operations in OSS is a high-risk operation. We recommend that you use a RAM user to call API operations or perform routine O&M. To create a RAM user, log on to the RAM console. 
    client, err := oss.New("yourEndpoint", "yourAccessKeyId", "yourAccessKeySecret")
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }    
    // Specify the name of the bucket. 
    bucket, err := client.Bucket("examplebucket")
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
    // Initiate a multipart upload task. 
    // Specify the full path of the object. Do not include the bucket name in the full path of the object. 
    imur, err := bucket.InitiateMultipartUpload("exampleobject.txt")
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
    // Cancel the multipart upload task. 
    err = bucket.AbortMultipartUpload(imur)
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
}

List uploaded parts

The following code provides an example on how to list the parts that are uploaded in a specific multipart upload task:

package main
import (
    "fmt"
    "os"
    "github.com/aliyun/aliyun-oss-go-sdk/oss"
)
func main() {
    // Create an OSSClient instance. 
    // Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. 
    // The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. Using these credentials to perform operations in OSS is a high-risk operation. We recommend that you use a RAM user to call API operations or perform routine O&M. To create a RAM user, log on to the RAM console. 
    client, err := oss.New("yourEndpoint", "yourAccessKeyId", "yourAccessKeySecret")
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
    // Specify the name of the bucket. 
    bucketName := "examplebucket"
    // Specify the full path of the object. Do not include the bucket name in the full path of the object. 
    objectName := "exampleobject.txt"
    // Specify the full path of the local file. By default, if you do not specify the full path of the local file, the local file is uploaded from the path of the project to which the sample program belongs. 
    locaFilename := "D:\\localpath\\examplefile.txt"
      
    bucket, err := client.Bucket(bucketName)
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
    // Split the local file into three parts. 
    chunks, err := oss.SplitFileByPartNum(locaFilename, 3)
    fd, err := os.Open(locaFilename)
    defer fd.Close()
    // Initiate a multipart upload task. 
    imur, err := bucket.InitiateMultipartUpload(objectName)
    uploadID := imur.UploadID
    fmt.Println("InitiateMultipartUpload UploadID: ", uploadID)
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
    // Upload the parts. 
    var parts []oss.UploadPart
    for _, chunk := range chunks {
        fd.Seek(chunk.Offset, os.SEEK_SET)
        // Call the UploadPart method to upload each part. 
        part, err := bucket.UploadPart(imur, fd, chunk.Size, chunk.Number)
        if err != nil {
            fmt.Println("Error:", err)
            os.Exit(-1)
        }
        fmt.Println("UploadPartNumber: ", part.PartNumber, ", ETag: ", part.ETag)
        parts = append(parts, part)
    }
    // Call the InitiateMultipartUploadResult method to list the uploaded parts. 
    lsRes, err := bucket.ListUploadedParts(imur)
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
    // Display the uploaded parts.  
    fmt.Println("\nParts:", lsRes.UploadedParts)
    for _, upload := range lsRes.UploadedParts {
        fmt.Println("List PartNumber:  ", upload.PartNumber, ", ETag: " ,upload.ETag, ", LastModified: ", upload.LastModified)
    }
    // List all uploaded parts based on the result that is obtained by using the InitiateMultipartUploadResult operation. The result is generated based on the specified object name and upload ID. 
    var imur_with_uploadid oss.InitiateMultipartUploadResult
    imur_with_uploadid.Key = objectName
    imur_with_uploadid.UploadID = uploadID
    // List the uploaded parts. 
    lsRes, err = bucket.ListUploadedParts(imur_with_uploadid)
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
    // Display the uploaded parts. 
    fmt.Println("\nListUploadedParts by UploadID: ", uploadID)
    for _, upload := range lsRes.UploadedParts {
        fmt.Println("List PartNumber:  ", upload.PartNumber, ", ETag: " ,upload.ETag, ", LastModified: ", upload.LastModified)
    }
    // Complete the multipart upload task. 
    cmur, err := bucket.CompleteMultipartUpload(imur, parts)
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
    fmt.Println("cmur:", cmur)
}

List multipart upload tasks

You can call the Bucket.ListMultipartUploads method to list all ongoing multipart upload tasks. Ongoing multipart upload tasks include tasks that have been initiated but are not completed or are canceled. The following table describes the parameters that you can configure to list multipart upload tasks.

• List multipart upload tasks by using default values of the parameters

  package main
  import (
      "fmt"
      "os"
      "github.com/aliyun/aliyun-oss-go-sdk/oss"
  )
  func main() {
      // Create an OSSClient instance. 
      // Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. 
      // The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. Using these credentials to perform operations in OSS is a high-risk operation. We recommend that you use a RAM user to call API operations or perform routine O&M. To create a RAM user, log on to the RAM console. 
      client, err := oss.New("yourEndpoint", "yourAccessKeyId", "yourAccessKeySecret")
      if err != nil {
          fmt.Println("Error:", err)
          os.Exit(-1)
      }    
      // Specify the name of the bucket. 
      bucketName := "examplebucket"
      bucket, err := client.Bucket(bucketName)
      if err != nil {
          fmt.Println("Error:", err)
          os.Exit(-1)
      }
      // List all multipart upload tasks. 
      keyMarker := ""
      uploadIdMarker := ""
      for {
          // By default, a maximum of 1,000 records can be returned at a time.  
          lsRes, err := bucket.ListMultipartUploads(oss.KeyMarker(keyMarker), oss.UploadIDMarker(uploadIdMarker))
          if err != nil {
              fmt.Println("Error:", err)
              os.Exit(-1)
          }
          // Display the multipart upload tasks.  
          for _, upload := range lsRes.Uploads {
              fmt.Println("Upload: ", upload.Key, ", UploadID: ",upload.UploadID)
          }
          if lsRes.IsTruncated {
              keyMarker = lsRes.NextKeyMarker
              uploadIdMarker = lsRes.NextUploadIDMarker
          } else {
              break
          }
      }
  }

• Set the Prefix parameter to file and list multipart upload tasks

      lsRes, err := bucket.ListMultipartUploads(oss.Prefix("file"))
      if err != nil {
          fmt.Println("Error:", err)
          os.Exit(-1)
      }
      fmt.Println("Uploads:", lsRes.Uploads)

• Specify that a maximum of 100 multipart upload tasks are returned

      lsRes, err := bucket.ListMultipartUploads(oss.MaxUploads(100))
      if err != nil {
          fmt.Println("Error:", err)
          os.Exit(-1)
      }
      fmt.Println("Uploads:", lsRes.Uploads)

• Set the Prefix parameter to file and specify that a maximum of 100 multipart upload tasks are returned

      lsRes, err := bucket.ListMultipartUploads(oss.Prefix("file"), oss.MaxUploads(100))
      if err != nil {
          fmt.Println("Error:", err)
          os.Exit(-1)
      }
      fmt.Println("Uploads:", lsRes.Uploads)

References

• For the complete sample code that is used to perform multipart upload, visit GitHub.
• The following API operations are required to perform multipart upload:
  • The API operation that you can call to initiate a multipart upload task. For more information, see InitiateMultipartUpload.
  • The API operation that you can call to upload data by part. For more information, see UploadPart.
  • The API operation that you can call to complete a multipart upload task. For more information, see CompleteMultipartUpload.
• For more information about the API operation that you can call to cancel a multipart upload task, see AbortMultipartUpload.
• For more information about the API operation that you can call to list the uploaded parts, see ListParts.
• For more information about the API operation that you can call to list ongoing multipart upload tasks, see ListMultipartUploads. Ongoing multipart upload tasks include tasks that have been initiated but are not completed or are canceled.