All Products
Search
Document Center

Object Storage Service:Multipart upload

Last Updated:Nov 01, 2023

Object Storage Service (OSS) provides the multipart upload feature. Multipart upload allows you to split a large object into multiple parts to upload. After these parts are uploaded, you can call the CompleteMultipartUpload operation to combine the parts into a complete object.

Usage notes

  • In this topic, the public endpoint of the China (Hangzhou) region is used. If you want to access OSS by using other Alibaba Cloud services in the same region as OSS, use an internal endpoint. For more information about the regions and endpoints supported by OSS, see Regions and endpoints.

  • In this topic, an OSSClient instance is created by using an OSS endpoint. If you want to create an OSSClient instance by using custom domain names or Security Token Service (STS), see Create an OSSClient instance.

  • To perform multipart upload, you must have the oss:PutObject permission. For more information, see Attach a custom policy to a RAM user.

Procedure

To upload an object by using multipart upload, perform the following steps:

  1. Initiate a multipart upload task.

    Call the InitiateMultipartUpload operation to obtain a unique upload ID in OSS.

  2. Upload parts.

    Call the UploadPart operation to upload the parts.

    Note
    • If parts are uploaded by a multipart upload task that has a specific upload ID, part numbers are used to identify the relative positions of the parts in an object. If you upload a part that has the same part number as an existing part, the existing part is overwritten by the uploaded part.

    • OSS includes the MD5 hash of each uploaded part in the ETag header in the response.

    • OSS calculates the MD5 hash of uploaded parts and compares the MD5 hash with the MD5 hash that is calculated by OSS SDK for Go. If the two hashes are different, OSS returns the InvalidDigest error code.

  3. Complete the multipart upload task.

    After all parts are uploaded, call the CompleteMultipartUpload operation to combine the parts into a complete object.

Sample code of multipart upload

The following sample code provides an example on how to implement a multipart upload task by following the multipart upload process:

#include <alibabacloud/oss/OssClient.h>
#include <fstream>

int64_t getFileSize(const std::string& file)
{
    std::fstream f(file, std::ios::in | std::ios::binary);
    f.seekg(0, f.end);
    int64_t size = f.tellg();
    f.close();
    return size;
}

using namespace AlibabaCloud::OSS;

int main(void)
{
    /* Initialize information about the account that is used to access OSS. */
    
    std::string Endpoint = "yourEndpoint";
    /* Specify the name of the bucket. Example: examplebucket. */
    std::string BucketName = "examplebucket";
    /* Specify the full path of the object. Do not include the bucket name in the full path of the object. Example: exampledir/exampleobject.txt.  */
    std::string ObjectName = "exampledir/exampleobject.txt";

    /* Initialize resources such as network resources. */
    InitializeSdk();

    ClientConfiguration conf;
    /* Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. */
    auto credentialsProvider = std::make_shared<EnvironmentVariableCredentialsProvider>();
    OssClient client(Endpoint, credentialsProvider, conf);
  
    InitiateMultipartUploadRequest initUploadRequest(BucketName, ObjectName);
    /* (Optional) Specify the storage class. */
    //initUploadRequest.MetaData().addHeader("x-oss-storage-class", "Standard");

    /* Initiate the multipart upload task. */
    auto uploadIdResult = client.InitiateMultipartUpload(initUploadRequest);
    /* Cancel the multipart upload task or list uploaded parts based on the upload ID. */
    /* If you want to cancel a multipart upload task based on the upload ID, obtain the upload ID after you call the InitiateMultipartUpload operation to initiate the multipart upload task. */
    /* If you want to list the uploaded parts in a multipart upload task based on the upload ID, obtain the upload ID after you call the InitiateMultipartUpload operation to initiate the multipart upload task but before you call the CompleteMultipartUpload operation to complete the multipart upload task. */ 
    auto uploadId = uploadIdResult.result().UploadId();
    std::string fileToUpload = "yourLocalFilename";
    int64_t partSize = 100 * 1024;
    PartList partETagList;
    auto fileSize = getFileSize(fileToUpload);
    int partCount = static_cast<int>(fileSize / partSize);
    /* Calculate the number of parts. */
    if (fileSize % partSize != 0) {
        partCount++;
    }

    /* Upload each part. */
    for (int i = 1; i <= partCount; i++) {
        auto skipBytes = partSize * (i - 1);
        auto size = (partSize < fileSize - skipBytes) ? partSize : (fileSize - skipBytes);
        std::shared_ptr<std::iostream> content = std::make_shared<std::fstream>(fileToUpload, std::ios::in|std::ios::binary);
        content->seekg(skipBytes, std::ios::beg);

        UploadPartRequest uploadPartRequest(BucketName, ObjectName, content);
        uploadPartRequest.setContentLength(size);
        uploadPartRequest.setUploadId(uploadId);
        uploadPartRequest.setPartNumber(i);
        auto uploadPartOutcome = client.UploadPart(uploadPartRequest);
        if (uploadPartOutcome.isSuccess()) {
            Part part(i, uploadPartOutcome.result().ETag());
            partETagList.push_back(part);
        }
        else {
            std::cout << "uploadPart fail" <<
            ",code:" << uploadPartOutcome.error().Code() <<
            ",message:" << uploadPartOutcome.error().Message() <<
            ",requestId:" << uploadPartOutcome.error().RequestId() << std::endl;
        }

    }

    /* Complete the multipart upload task. */
    /* When the multipart upload task is completed, you need to provide all valid PartETags. After OSS receives the PartETags, OSS verifies all parts one by one. After part verification is successful, OSS combines these parts into a complete object. */
    CompleteMultipartUploadRequest request(BucketName, ObjectName);
    request.setUploadId(uploadId);
    request.setPartList(partETagList);
    /* (Optional) Specify the ACL of the object. */
    //request.setAcl(CannedAccessControlList::Private);

    auto outcome = client.CompleteMultipartUpload(request);

    if (!outcome.isSuccess()) {
        /* Handle exceptions. */
        std::cout << "CompleteMultipartUpload fail" <<
        ",code:" << outcome.error().Code() <<
        ",message:" << outcome.error().Message() <<
        ",requestId:" << outcome.error().RequestId() << std::endl;
        return -1;
    }

    /* Release resources such as network resources. */
    ShutdownSdk();
    return 0;
}

To call the CompleteMultipartUpload operation when you obtain uploaded parts, you must provide the ETag value of each part. You can obtain the ETag value of each part by using one of the following methods:

  • When a part is uploaded, the ETag value of the part is included in the response. You can save and use this ETag value. This method is used in the preceding sample code.

  • Call the ListParts operation to query the ETag values of the uploaded parts.

List uploaded parts

You can call the ListParts method to list all uploaded parts that use the specified upload ID.

Note

By default, you can list up to 1,000 uploaded parts by using simple list. If the number of uploaded parts exceeds 1,000, list all uploaded parts by page.

List uploaded parts by using simple list

The following sample code provides an example on how to use simple list to list uploaded parts:

#include <alibabacloud/oss/OssClient.h>
using namespace AlibabaCloud::OSS;

int main(void)
{
    /* Initialize information about the account that is used to access OSS. */
    
    std::string Endpoint = "yourEndpoint";
    /* Specify the name of the bucket. Example: examplebucket. */
    std::string BucketName = "examplebucket";
    /* Specify the full path of the object. Do not include the bucket name in the full path of the object. Example: exampledir/exampleobject.txt. */
    std::string ObjectName = "exampledir/exampleobject.txt";
    /* Specify the upload ID. You can obtain the upload ID from the response to the InitiateMultipartUpload operation. You must obtain the upload ID before you call the CompleteMultipartUpload operation to complete the multipart upload task. */
    std::string UploadId = "yourUploadId";

    /* Initialize resources such as network resources. */
    InitializeSdk();

    ClientConfiguration conf;
    /* Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. */
    auto credentialsProvider = std::make_shared<EnvironmentVariableCredentialsProvider>();
    OssClient client(Endpoint, credentialsProvider, conf);

    /* List the uploaded parts. By default, 1,000 parts are listed. */
    ListPartsRequest listuploadrequest(BucketName, ObjectName);
    listuploadrequest.setUploadId(UploadId);

    bool IsTruncated = false;

    do {
        auto listUploadResult = client.ListParts(listuploadrequest);
        if (!listUploadResult.isSuccess()) {
            /* Handle exceptions. */
            std::cout << "ListParts fail" <<
            ",code:" << listUploadResult.error().Code() <<
            ",message:" << listUploadResult.error().Message() <<
            ",requestId:" << listUploadResult.error().RequestId() << std::endl;
            break;
        }
        else {
            for (const auto& part : listUploadResult.result().PartList()) {
                std::cout << "part"<<
                ",name:" << part.PartNumber() <<
                ",size:" << part.Size() <<
                ",etag:" << part.ETag() <<
                ",lastmodify time:" << part.LastModified() << std::endl;
            }
        }
        listuploadrequest.setPartNumberMarker(listUploadResult.result().NextPartNumberMarker());
        IsTruncated = listUploadResult.result().IsTruncated();
    } while (IsTruncated);

    /* Release resources such as network resources. */
    ShutdownSdk();
    return 0;
}

List all uploaded parts by page

The following sample code provides an example on how to specify the maximum number of parts to list on each page and list all parts by page:

#include <alibabacloud/oss/OssClient.h>
using namespace AlibabaCloud::OSS;

int main(void)
{
    /* Initialize information about the account that is used to access OSS. */
    
    std::string Endpoint = "yourEndpoint";
    /* Specify the name of the bucket. Example: examplebucket. */
    std::string BucketName = "examplebucket";
    /* Specify the full path of the object. Do not include the bucket name in the full path of the object. Example: exampledir/exampleobject.txt.  */
    std::string ObjectName = "exampledir/exampleobject.txt";
    /* Specify the upload ID. You can obtain the upload ID from the response to the InitiateMultipartUpload operation. You must obtain the upload ID before you call the CompleteMultipartUpload operation to complete the multipart upload task. */
    std::string UploadId = "yourUploadId";

    /* Initialize resources such as network resources. */
    InitializeSdk();

    ClientConfiguration conf;
    /* Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. */
    auto credentialsProvider = std::make_shared<EnvironmentVariableCredentialsProvider>();
    OssClient client(Endpoint, credentialsProvider, conf);

    /* List all uploaded parts by page. */
    /* Set the maximum number of parts to list on each page. */
    ListPartsRequest listuploadrequest(BucketName, ObjectName);
    listuploadrequest.setMaxParts(50);
    listuploadrequest.setUploadId(UploadId);
    bool IsTruncated = false;
    do {
        auto listuploadresult = client.ListParts(listuploadrequest);
        if (!listuploadresult.isSuccess()) {
            /* Handle exceptions. */
            std::cout << "ListParts fail" <<
            ",code:" << listuploadresult.error().Code() <<
            ",message:" << listuploadresult.error().Message() <<
            ",requestId:" << listuploadresult.error().RequestId() << std::endl;
            break;
        }
        else {
            for (const auto& part : listuploadresult.result().PartList()) {
                std::cout << "part"<<
                ",name:" << part.PartNumber() <<
                ",size:" << part.Size() <<
                ",etag:" << part.ETag() <<
                ",lastmodify time:" << part.LastModified() << std::endl;
            }
        }  
        listuploadrequest.setPartNumberMarker(listuploadresult.result().NextPartNumberMarker());
        IsTruncated = listuploadresult.result().IsTruncated();
    } while (IsTruncated);

    /* Release resources such as network resources. */
    ShutdownSdk();
    return 0;
}

List multipart upload tasks

You can call the ListMultipartUploads method to list all ongoing multipart upload tasks. Ongoing multipart upload tasks are tasks that have been initiated but are not completed or canceled.

Note

By default, you can list up to 1,000 multipart upload tasks by using simple list. If the number of multipart upload tasks exceeds 1,000, list all multipart upload tasks by page.

List multipart upload tasks by using simple list

The following sample code provides an example on how to use simple list to list multipart upload tasks:

#include <alibabacloud/oss/OssClient.h>
using namespace AlibabaCloud::OSS;

int main(void)
{
    /* Initialize information about the account that is used to access OSS. */
    
    std::string Endpoint = "yourEndpoint";
    /* Specify the name of the bucket. Example: examplebucket. */
    std::string BucketName = "examplebucket";    

    /* Initialize resources such as network resources. */
    InitializeSdk();

    ClientConfiguration conf;
    /* Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. */
    auto credentialsProvider = std::make_shared<EnvironmentVariableCredentialsProvider>();
    OssClient client(Endpoint, credentialsProvider, conf);

    /* List the completed multipart upload tasks. By default, 1,000 parts are listed. */
    ListMultipartUploadsRequest listmultiuploadrequest(BucketName);
	bool IsTruncated = false;
    do {
        auto listresult = client.ListMultipartUploads(listmultiuploadrequest);
        if (!listresult.isSuccess()) {
            /* Handle exceptions. */
            std::cout << "ListMultipartUploads fail" <<
            ",code:" << listresult.error().Code() <<
            ",message:" << listresult.error().Message() <<
            ",requestId:" << listresult.error().RequestId() << std::endl;
            break;
        }
        else {
            for (const auto& part : listresult.result().MultipartUploadList()) {
                std::cout << "part"<<
                ",name:" << part.Key <<
                ",uploadid:" << part.UploadId <<
                ",initiated time:" << part.Initiated << std::endl;
            }
        }
        listmultiuploadrequest.setKeyMarker(listresult.result().NextKeyMarker()); 
        listmultiuploadrequest.setUploadIdMarker(listresult.result().NextUploadIdMarker());
        IsTruncated = listresult.result().IsTruncated();
    } while (IsTruncated);

    /* Release resources such as network resources. */
    ShutdownSdk();
    return 0;
}

List all multipart upload tasks by page

The following sample code provides an example on how to list all multipart upload tasks by page:

#include <alibabacloud/oss/OssClient.h>
using namespace AlibabaCloud::OSS;

int main(void)
{
    /* Initialize information about the account that is used to access OSS. */
    
    std::string Endpoint = "yourEndpoint";
    /* Specify the name of the bucket. Example: examplebucket. */
    std::string BucketName = "examplebucket"; 

    /* Initialize resources such as network resources. */
    InitializeSdk();

    ClientConfiguration conf;
    /* Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. */
    auto credentialsProvider = std::make_shared<EnvironmentVariableCredentialsProvider>();
    OssClient client(Endpoint, credentialsProvider, conf);

    /* List all multipart upload tasks by page. */
    /* Set the maximum number of multipart upload tasks to list on each page. */
    ListMultipartUploadsRequest  listmultiuploadrequest(BucketName);
    listmultiuploadrequest.setMaxUploads(50);
    bool IsTruncated = false;
    do {
        auto listresult = client.ListMultipartUploads(listmultiuploadrequest);
        if (!listresult.isSuccess()) {
            /* Handle exceptions. */
            std::cout << "ListMultipartUploads fail" <<
            ",code:" << listresult.error().Code() <<
            ",message:" << listresult.error().Message() <<
            ",requestId:" << listresult.error().RequestId() << std::endl;
            break;
        }
        else {
            for (const auto& part : listresult.result().MultipartUploadList()) {
                std::cout << "part"<<
                ",name:" << part.Key <<
                ",uploadid:" << part.UploadId <<
                ",initiated time:" << part.Initiated << std::endl;
            }
        }  
        listmultiuploadrequest.setKeyMarker(listresult.result().NextKeyMarker()); 
        listmultiuploadrequest.setUploadIdMarker(listresult.result().NextUploadIdMarker());
        IsTruncated = listresult.result().IsTruncated();
    } while (IsTruncated);

    /* Release resources such as network resources. */
    ShutdownSdk();
    return 0;
}

Cancel a multipart upload task

You can call the client.AbortMultipartUpload method to cancel a multipart upload task. If a multipart upload task is canceled, the upload ID cannot be used to upload parts. In addition, the uploaded parts are deleted.

The following sample code provides an example on how to cancel a multipart upload task:

#include <alibabacloud/oss/OssClient.h>
using namespace AlibabaCloud::OSS;

int main(void)
{
    /* Initialize information about the account that is used to access OSS. */
    
    std::string Endpoint = "yourEndpoint";
    /* Specify the name of the bucket. Example: examplebucket. */
    std::string BucketName = "examplebucket";
    /* Specify the full path of the object. Do not include the bucket name in the full path of the object. Example: exampledir/exampleobject.txt.  */
    std::string ObjectName = "exampledir/exampleobject.txt";
    /* Specify the upload ID. You can obtain the upload ID from the response to the InitiateMultipartUpload operation. */
    std::string UploadId = "yourUploadId";

    /* Initialize resources such as network resources. */
    InitializeSdk();

    ClientConfiguration conf;
    /* Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. */
    auto credentialsProvider = std::make_shared<EnvironmentVariableCredentialsProvider>();
    OssClient client(Endpoint, credentialsProvider, conf);
  
    InitiateMultipartUploadRequest initUploadRequest(BucketName, ObjectName);

    /* Initiate the multipart upload task. */
    auto uploadIdResult = client.InitiateMultipartUpload(initUploadRequest);
    auto uploadId = uploadIdResult.result().UploadId();

    /* Cancel the multipart upload task. */
    AbortMultipartUploadRequest  abortUploadRequest(BucketName, ObjectName, uploadId);
    auto outcome = client.AbortMultipartUpload(abortUploadRequest);

    if (!outcome.isSuccess()) {
        /* Handle exceptions. */
        std::cout << "AbortMultipartUpload fail" <<
        ",code:" << outcome.error().Code() <<
        ",message:" << outcome.error().Message() <<
        ",requestId:" << outcome.error().RequestId() << std::endl;
        return -1;
    }

    /* Release resources such as network resources. */
    ShutdownSdk();
    return 0;
}

References

  • For the complete sample code that is used to perform multipart upload, visit GitHub.

  • A multipart upload involves three API operations. For more information about the operations, see the following topics:

  • For more information about the API operation that you can call to cancel a multipart upload task, see AbortMultipartUpload.

  • For more information about the API operation that you can call to list uploaded parts, see ListParts.

  • For more information about the API operation that you can call to list all ongoing multipart upload tasks, see ListMultipartUploads. Ongoing multipart upload tasks are tasks that have been initiated but are not completed or canceled.