Object Storage Service (OSS) provides the multipart upload feature. Multipart upload allows you to split a large object into multiple parts to upload. After these parts are uploaded, you can call CompleteMultipartUpload to combine the parts into a complete object to implement resumable upload.
Usage notes
- In this topic, the public endpoint of the China (Hangzhou) region is used. If you want to access OSS by using other Alibaba Cloud services in the same region as OSS, use an internal endpoint. For more information about the regions and endpoints supported by OSS, see Regions and endpoints.
- In this topic, an OSSClient instance is created by using an OSS endpoint. If you want to create an OSSClient by using custom domain names or STS, see Initialization.
- The
oss:PutObject
permission is required to perform multipart upload. For more information, see Attach a custom policy to a RAM user.
Multipart upload process
To implement multipart upload, perform the following operations:
- Initiate a multipart upload task.
Call the Bucket.InitiateMultipartUpload method to obtain a unique upload ID in OSS.
- Upload parts.
Call the Bucket.UploadPart method to upload the parts.
Note- For parts that are uploaded by a multipart upload task with a specific upload ID, part numbers identify their relative positions in an object. If you upload a part and use its part number again to upload another part, the latter part overwrites the former part.
- OSS includes the MD5 hash of part data in the ETag header and returns the MD5 hash to the user.
- OSS calculates the MD5 hash of uploaded data and compares the MD5 hash with the MD5 hash calculated by the SDK. If the two hashes are different, the InvalidDigest error code is returned.
- Complete the multipart upload task.
After all parts are uploaded, call the Bucket.CompleteMultipartUpload method to combine the parts into a complete object.
Complete sample code
The following code provides an example on how to perform multipart upload:
package main
import (
"fmt"
"os"
"github.com/aliyun/aliyun-oss-go-sdk/oss"
)
func main() {
// Create an OSSClient instance.
// Set yourEndpoint to the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set yourEndpoint to https://oss-cn-hangzhou.aliyuncs.com.
// Security risks may arise if you use the AccessKey pair of an Alibaba Cloud account to access OSS because the account has permissions on all API operations. We recommend that you use a RAM user to call API operations or perform routine operations and maintenance. To create a RAM user, log on to the RAM console.
client, err := oss.New("yourEndpoint", "yourAccessKeyId", "yourAccessKeySecret")
if err != nil {
fmt.Println("Error:", err)
os.Exit(-1)
}
// Specify the name of the bucket.
bucketName := "examplebucket"
// Specify the full path of the object. The full path of the object cannot contain the bucket name.
objectName := "exampleobject.txt"
// Specify the full path of the local file that you want to upload. By default, if you do not specify the full path of the local file, the local file is uploaded from the path of the project to which the sample program belongs.
locaFilename := "D:\\localpath\\examplefile.txt"
bucket, err := client.Bucket(bucketName)
if err != nil {
fmt.Println("Error:", err)
os.Exit(-1)
}
// Split the local file into three parts.
chunks, err := oss.SplitFileByPartNum(locaFilename, 3)
fd, err := os.Open(locaFilename)
defer fd.Close()
// Specify the expiration time.
expires := time.Date(2049, time.January, 10, 23, 0, 0, 0, time.UTC)
// The following code provides an example on how to specify the request headers when you initiate a multipart upload task.
options := []oss.Option{
oss.MetadataDirective(oss.MetaReplace),
oss.Expires(expires),
// Specify the caching behavior of the web page when the object is downloaded.
// oss.CacheControl("no-cache"),
// Specify the name of the object when the object is downloaded.
// oss.ContentDisposition("attachment;filename=FileName.txt"),
// Specify the encoding format for the content of the object.
// oss.ContentEncoding("gzip"),
// Specify the method that is used to encode the object name in the response. Only URL encoding is supported.
// oss.EncodingType("url"),
// Specify the storage class of the object.
// oss.ObjectStorageClass(oss.StorageStandard),
}
// Step 1: Initiate a multipart upload task and set the storage class to Standard.
imur, err := bucket.InitiateMultipartUpload(objectName, options...)
// Step 2: Upload parts.
var parts []oss.UploadPart
for _, chunk := range chunks {
fd.Seek(chunk.Offset, os.SEEK_SET)
// Call the UploadPart method to upload each part.
part, err := bucket.UploadPart(imur, fd, chunk.Size, chunk.Number)
if err != nil {
fmt.Println("Error:", err)
os.Exit(-1)
}
parts = append(parts, part)
}
// Set the access control list (ACL) of the object to public-read. By default, the object inherits the ACL of the bucket.
objectAcl := oss.ObjectACL(oss.ACLPublicRead)
// Step 3: Complete the multipart upload task and set the ACL of the object to public-read.
cmur, err := bucket.CompleteMultipartUpload(imur, parts, objectAcl)
if err != nil {
fmt.Println("Error:", err)
os.Exit(-1)
}
fmt.Println("cmur:", cmur)
}
Cancel a multipart upload task
You can call the bucket.AbortMultipartUpload method to cancel a multipart upload task. If you cancel a multipart upload task, you cannot use the upload ID to manage parts. The uploaded parts are deleted.
The following code provides an example on how to cancel a multipart upload task:
package main
import (
"fmt"
"os"
"github.com/aliyun/aliyun-oss-go-sdk/oss"
)
func main() {
// Create an OSSClient instance.
// Set yourEndpoint to the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set yourEndpoint to https://oss-cn-hangzhou.aliyuncs.com.
// Security risks may arise if you use the AccessKey pair of an Alibaba Cloud account to access OSS because the account has permissions on all API operations. We recommend that you use a RAM user to call API operations or perform routine operations and maintenance. To create a RAM user, log on to the RAM console.
client, err := oss.New("yourEndpoint", "yourAccessKeyId", "yourAccessKeySecret")
if err != nil {
fmt.Println("Error:", err)
os.Exit(-1)
}
// Specify the bucket name.
bucket, err := client.Bucket("examplebucket")
if err != nil {
fmt.Println("Error:", err)
os.Exit(-1)
}
// Initiate a multipart upload task.
// Specify the full path of the object. The full path of the object cannot contain the bucket name.
imur, err := bucket.InitiateMultipartUpload("exampleobject.txt")
if err != nil {
fmt.Println("Error:", err)
os.Exit(-1)
}
// Cancel the multipart upload task.
err = bucket.AbortMultipartUpload(imur)
if err != nil {
fmt.Println("Error:", err)
os.Exit(-1)
}
}
List the uploaded parts
The following code provides an example on how to list the parts that are uploaded in the specified multipart upload task:
package main
import (
"fmt"
"os"
"github.com/aliyun/aliyun-oss-go-sdk/oss"
)
func main() {
// Create an OSSClient instance.
// Set yourEndpoint to the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set yourEndpoint to https://oss-cn-hangzhou.aliyuncs.com.
// Security risks may arise if you use the AccessKey pair of an Alibaba Cloud account to access OSS because the account has permissions on all API operations. We recommend that you use a RAM user to call API operations or perform routine operations and maintenance. To create a RAM user, log on to the RAM console.
client, err := oss.New("yourEndpoint", "yourAccessKeyId", "yourAccessKeySecret")
if err != nil {
fmt.Println("Error:", err)
os.Exit(-1)
}
// Specify the bucket name.
bucketName := "examplebucket"
// Specify the full path of the object. The full path of the object cannot contain the bucket name.
objectName := "exampleobject.txt"
// Specify the full path of the local file that you want to upload. By default, if you do not specify the local file, the local file is uploaded from the path of the project to which the sample program belongs.
locaFilename := "D:\\localpath\\examplefile.txt"
bucket, err := client.Bucket(bucketName)
if err != nil {
fmt.Println("Error:", err)
os.Exit(-1)
}
// Split the local file into three parts.
chunks, err := oss.SplitFileByPartNum(locaFilename, 3)
fd, err := os.Open(locaFilename)
defer fd.Close()
// Initiate a multipart upload task.
imur, err := bucket.InitiateMultipartUpload(objectName)
uploadID := imur.UploadID
fmt.Println("InitiateMultipartUpload UploadID: ", uploadID)
if err != nil {
fmt.Println("Error:", err)
os.Exit(-1)
}
// Upload parts.
var parts []oss.UploadPart
for _, chunk := range chunks {
fd.Seek(chunk.Offset, os.SEEK_SET)
// Call the UploadPart method to upload each part.
part, err := bucket.UploadPart(imur, fd, chunk.Size, chunk.Number)
if err != nil {
fmt.Println("Error:", err)
os.Exit(-1)
}
fmt.Println("UploadPartNumber: ", part.PartNumber, ", ETag: ", part.ETag)
parts = append(parts, part)
}
// List the uploaded parts based on InitiateMultipartUploadResult.
lsRes, err := bucket.ListUploadedParts(imur)
if err != nil {
fmt.Println("Error:", err)
os.Exit(-1)
}
// Display the uploaded parts.
fmt.Println("\nParts:", lsRes.UploadedParts)
for _, upload := range lsRes.UploadedParts {
fmt.Println("List PartNumber: ", upload.PartNumber, ", ETag: " ,upload.ETag, ", LastModified: ", upload.LastModified)
}
// Specify objectName and uploadID to obtain InitiateMultipartUploadResult. List all uploaded parts based on the result.
var imur_with_uploadid oss.InitiateMultipartUploadResult
imur_with_uploadid.Key = objectName
imur_with_uploadid.UploadID = uploadID
// List the uploaded parts.
lsRes, err = bucket.ListUploadedParts(imur_with_uploadid)
if err != nil {
fmt.Println("Error:", err)
os.Exit(-1)
}
// Display the uploaded parts.
fmt.Println("\nListUploadedParts by UploadID: ", uploadID)
for _, upload := range lsRes.UploadedParts {
fmt.Println("List PartNumber: ", upload.PartNumber, ", ETag: " ,upload.ETag, ", LastModified: ", upload.LastModified)
}
// Complete the multipart upload task.
cmur, err := bucket.CompleteMultipartUpload(imur, parts)
if err != nil {
fmt.Println("Error:", err)
os.Exit(-1)
}
fmt.Println("cmur:", cmur)
}
List multipart upload tasks
You can call the Bucket.ListMultipartUploads
method to list all ongoing multipart upload tasks, which include tasks that have
been initiated but are not completed and tasks that are canceled. The following table
describes the parameters that you can configure to list these tasks.
Parameter | Description |
---|---|
Delimiter | The character that is used to group objects by name. Objects whose names contain the same string that ranges from the specified prefix to the delimiter that appears for the first time are grouped as an element. |
MaxUploads | The maximum number of multipart upload tasks to return this time. Default value: 1000. Maximum value: 1000. |
KeyMarker | Specifies that all multipart upload tasks with objects whose names are alphabetically after the value of the KeyMarker parameter are included in the list. You can use this parameter with the UploadIDMarker parameter to specify the start position to list the returned results. |
Prefix | The prefix that the returned object names must contain. If you use a prefix for a query, the returned object name contains the prefix. |
UploadIDMarker | The start position from which to list the returned results. This parameter is used
together with the KeyMarker parameter.
|
- List multipart upload tasks by using default parameter values
package main import ( "fmt" "os" "github.com/aliyun/aliyun-oss-go-sdk/oss" ) func main() { // Create an OSSClient instance. // Set yourEndpoint to the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set yourEndpoint to https://oss-cn-hangzhou.aliyuncs.com. // Security risks may arise if you use the AccessKey pair of an Alibaba Cloud account to access OSS because the account has permissions on all API operations. We recommend that you use a RAM user to call API operations or perform routine operations and maintenance. To create a RAM user, log on to the RAM console. client, err := oss.New("yourEndpoint", "yourAccessKeyId", "yourAccessKeySecret") if err != nil { fmt.Println("Error:", err) os.Exit(-1) } // Specify the bucket name. bucketName := "examplebucket" bucket, err := client.Bucket(bucketName) if err != nil { fmt.Println("Error:", err) os.Exit(-1) } // List all multipart upload tasks. keyMarker := "" uploadIdMarker := "" for { // By default, a maximum of 1,000 records can be returned at a time. lsRes, err := bucket.ListMultipartUploads(oss.KeyMarker(keyMarker), oss.UploadIDMarker(uploadIdMarker)) if err != nil { fmt.Println("Error:", err) os.Exit(-1) } // Display the multipart upload tasks. for _, upload := range lsRes.Uploads { fmt.Println("Upload: ", upload.Key, ", UploadID: ",upload.UploadID) } if lsRes.IsTruncated { keyMarker = lsRes.NextKeyMarker uploadIdMarker = lsRes.NextUploadIDMarker } else { break } } }
- Specify file as the prefix
lsRes, err := bucket.ListMultipartUploads(oss.Prefix("file")) if err != nil { fmt.Println("Error:", err) os.Exit(-1) } fmt.Println("Uploads:", lsRes.Uploads)
- Specify that a maximum of 100 multipart upload tasks are returned
lsRes, err := bucket.ListMultipartUploads(oss.MaxUploads(100)) if err != nil { fmt.Println("Error:", err) os.Exit(-1) } fmt.Println("Uploads:", lsRes.Uploads)
- Specify file as the prefix and specify that a maximum of 100 multipart upload tasks
are returned
lsRes, err := bucket.ListMultipartUploads(oss.Prefix("file"), oss.MaxUploads(100)) if err != nil { fmt.Println("Error:", err) os.Exit(-1) } fmt.Println("Uploads:", lsRes.Uploads)
References
- For more information about the complete sample code of multipart upload, visit GitHub.
- The following API operations are required to perform multipart upload:
- The API operation that you can call to initiate a multipart upload task. For more information, see InitiateMultipartUpload.
- The API operation that you can call to upload data by part. For more information, see UploadPart.
- The API operation that you can call to complete a multipart upload task. For more information, see CompleteMultipartUpload.
- For more information about the API operation that you can call to cancel a multipart upload task, see AbortMultipartUpload.
- For more information about the API operation that you can call to list the uploaded parts, see ListParts.
- For more information about the API operation that you can call to list all ongoing multipart upload tasks, see ListMultipartUploads. Ongoing multipart upload tasks include tasks that have been initiated but are not completed and tasks that are canceled.