Object Storage Service:Multipart upload

Usage notes

• In this topic, the public endpoint of the China (Hangzhou) region is used. If you want to access OSS by using other Alibaba Cloud services in the same region as OSS, use an internal endpoint For more information about the regions and endpoints supported by OSS, see Regions and endpoints.
• In this topic, an OSSClient instance is created by using an OSS endpoint. If you want to create an OSSClient instance by using custom domain names or STS, see Create an OSSClient instance.
• The oss:PutObject permission is required to perform multipart upload. For more information, see Attach a custom policy to a RAM user.

Process

To implement multipart upload, perform the following steps:

1. Initiate a multipart upload task.

   Call the InitiateMultipartUpload operation to obtain a unique upload ID in OSS.

2. Upload parts.

   Call the UploadPart operation to upload the parts.

   Note

   • If parts are uploaded by a multipart upload task that has a specific upload ID, part numbers are used to identify the relative positions of the parts in an object. If you upload a part that has the same part number as an existing part, the existing part is overwritten by the uploaded part.
   • OSS includes the MD5 hash of each uploaded part in the ETag header in the response.
   • OSS calculates the MD5 hash of uploaded data and compares the MD5 hash with the MD5 hash that is calculated by OSS SDK for Java. If the two hashes are different, OSS returns the InvalidDigest error code.
3. Complete the multipart upload task.

   After all parts are uploaded, call the CompleteMultipartUpload operation to combine the parts into a complete object.

Examples

The following code provides an example on how to perform multipart upload:

#include <alibabacloud/oss/OssClient.h>

int64_t getFileSize(const std::string& file)
{
    std::fstream f(file, std::ios::in | std::ios::binary);
    f.seekg(0, f.end);
    int64_t size = f.tellg();
    f.close();
    return size;
}

using namespace AlibabaCloud::OSS;

int main(void)
{
    /* Initialize the information about the account used to access OSS. */
    std::string AccessKeyId = "yourAccessKeyId";
    std::string AccessKeySecret = "yourAccessKeySecret";
    std::string Endpoint = "yourEndpoint";
    /* Specify the name of the bucket. Example: examplebucket. */
    std::string BucketName = "examplebucket";
    /* Specify the full path of the object. Do not include the bucket name in the full path. Example: exampledir/exampleobject.txt.  */
    std::string ObjectName = "exampledir/exampleobject.txt";

    /* Initialize resources, such as network resources. */
    InitializeSdk();

    ClientConfiguration conf;
    OssClient client(Endpoint, AccessKeyId, AccessKeySecret, conf);
    InitiateMultipartUploadRequest initUploadRequest(BucketName, ObjectName);
    /* (Optional) Specify the storage class. */
    //initUploadRequest.MetaData().addHeader("x-oss-storage-class", "Standard");

    /* Initiate the multipart upload task. */
    auto uploadIdResult = client.InitiateMultipartUpload(initUploadRequest);
    auto uploadId = uploadIdResult.result().UploadId();
    std::string fileToUpload = "yourLocalFilename";
    int64_t partSize = 100 * 1024;
    PartList partETagList;
    auto fileSize = getFileSize(fileToUpload);
    int partCount = static_cast<int>(fileSize / partSize);
    /* Calculate the number of parts. */
    if (fileSize % partSize != 0) {
        partCount++;
    }

    /* Upload each part. */
    for (int i = 1; i <= partCount; i++) {
        auto skipBytes = partSize * (i - 1);
        auto size = (partSize < fileSize - skipBytes) ? partSize : (fileSize - skipBytes);
        std::shared_ptr<std::iostream> content = std::make_shared<std::fstream>(fileToUpload, std::ios::in|std::ios::binary);
        content->seekg(skipBytes, std::ios::beg);

        UploadPartRequest uploadPartRequest(BucketName, ObjectName, content);
        uploadPartRequest.setContentLength(size);
        uploadPartRequest.setUploadId(uploadId);
        uploadPartRequest.setPartNumber(i);
        auto uploadPartOutcome = client.UploadPart(uploadPartRequest);
        if (uploadPartOutcome.isSuccess()) {
            Part part(i, uploadPartOutcome.result().ETag());
            partETagList.push_back(part);
        }
        else {
            std::cout << "uploadPart fail" <<
            ",code:" << uploadPartOutcome.error().Code() <<
            ",message:" << uploadPartOutcome.error().Message() <<
            ",requestId:" << uploadPartOutcome.error().RequestId() << std::endl;
        }

    }

    /* Complete the multipart upload task. */
    CompleteMultipartUploadRequest request(BucketName, ObjectName);
    request.setUploadId(uploadId);
    request.setPartList(partETagList);
    /* (Optional) Specify the ACL of the object. */
    //request.setAcl(CannedAccessControlList::Private);

    auto outcome = client.CompleteMultipartUpload(request);

    if (!outcome.isSuccess()) {
        /* Handle exceptions. */
        std::cout << "CompleteMultipartUpload fail" <<
        ",code:" << outcome.error().Code() <<
        ",message:" << outcome.error().Message() <<
        ",requestId:" << outcome.error().RequestId() << std::endl;
        return -1;
    }

    /* Release resources, such as network resources. */
    ShutdownSdk();
    return 0;
}

To call the CompleteMultipartUpload operation when you obtain uploaded parts, you must provide the ETag value of each part. You can obtain the ETag value of each part by using one of the following methods:

• When a part is uploaded, the ETag value of the part is included in the response. You can save and use this ETag value. This method is used in the preceding sample code.
• Call the ListParts operation to query the ETag values of the uploaded parts.

List uploaded parts

List multipart upload tasks

You can call the ListMultipartUploads operation to list all ongoing multipart upload tasks. Ongoing multipart upload tasks are tasks that have been initiated but are not completed or canceled.

• List all multipart upload tasks
  Note By default, the ListMultipartUploads operation can list up to 1,000 multipart upload tasks at a time. To list more than 1,000 multipart upload tasks, you must list the results by page in the multipart upload operation.

  The following sample code provides an example on how to list all multipart upload tasks:

  #include <alibabacloud/oss/OssClient.h>
  using namespace AlibabaCloud::OSS;
  
  int main(void)
  {
      /* Initialize the information about the account used to access OSS. */
      std::string AccessKeyId = "yourAccessKeyId";
      std::string AccessKeySecret = "yourAccessKeySecret";
      std::string Endpoint = "yourEndpoint";
      /* Specify the name of the bucket. Example: examplebucket. */
      std::string BucketName = "examplebucket";    
  
      /* Initialize resources, such as network resources. */
      InitializeSdk();
  
      ClientConfiguration conf;
      OssClient client(Endpoint, AccessKeyId, AccessKeySecret, conf);
  
      /* List the completed multipart upload tasks. By default, 1,000 parts are listed. */
      ListMultipartUploadsRequest listmultiuploadrequest(BucketName);
      do {
          auto listresult = client.ListMultipartUploads(listmultiuploadrequest);
          if (!listresult.isSuccess()) {
              /* Handle exceptions. */
              std::cout << "ListMultipartUploads fail" <<
              ",code:" << listresult.error().Code() <<
              ",message:" << listresult.error().Message() <<
              ",requestId:" << listresult.error().RequestId() << std::endl;
              break;
          }
          else {
              for (const auto& part : listresult.result().MultipartUploadList()) {
                  std::cout << "part"<<
                  ",name:" << part.Key <<
                  ",uploadid:" << part.UploadId <<
                  ",initiated time:" << part.Initiated << std::endl;
              }
          }
          listmultiuploadrequest.setKeyMarker(listresult.result().NextKeyMarker()); 
          listmultiuploadrequest.setUploadIdMarker(listresult.result().NextUploadIdMarker()); 
      } while (listresult.result().IsTruncated());
  
      /* Release resources, such as network resources. */
      ShutdownSdk();
      return 0;
  }

• List all multipart upload tasks by page
  The following sample code provides an example on how to list all multipart upload tasks by page:

  #include <alibabacloud/oss/OssClient.h>
  using namespace AlibabaCloud::OSS;
  
  int main(void)
  {
      /* Initialize the information about the account used to access OSS. */
      std::string AccessKeyId = "yourAccessKeyId";
      std::string AccessKeySecret = "yourAccessKeySecret";
      std::string Endpoint = "yourEndpoint";
      /* Specify the name of the bucket. Example: examplebucket. */
      std::string BucketName = "examplebucket"; 
  
      /* Initialize resources, such as network resources. */
      InitializeSdk();
  
      ClientConfiguration conf;
      OssClient client(Endpoint, AccessKeyId, AccessKeySecret, conf);
  
      /* List all multipart upload tasks by page. */
      /* Set the maximum number of multipart upload tasks to list on each page. */
      ListMultipartUploadsRequest  listmultiuploadrequest(BucketName);
      listmultiuploadrequest.setMaxUploads(50);
      do {
          listresult = client.ListMultipartUploads(listmultiuploadrequest);
          if (!listresult.isSuccess()) {
              /* Handle exceptions. */
              std::cout << "ListMultipartUploads fail" <<
              ",code:" << listresult.error().Code() <<
              ",message:" << listresult.error().Message() <<
              ",requestId:" << listresult.error().RequestId() << std::endl;
              break;
          }
          else {
              for (const auto& part : listresult.result().MultipartUploadList()) {
                  std::cout << "part"<<
                  ",name:" << part.Key <<
                  ",uploadid:" << part.UploadId <<
                  ",initiated time:" << part.Initiated << std::endl;
              }
          }  
          listmultiuploadrequest.setKeyMarker(listresult.result().NextKeyMarker()); 
          listmultiuploadrequest.setUploadIdMarker(listresult.result().NextUploadIdMarker()); 
      } while (listresult.result().IsTruncated());
  
      /* Release resources, such as network resources. */
      ShutdownSdk();
      return 0;
  }

Cancel a multipart upload task

You can call the client.AbortMultipartUpload method to cancel a multipart upload task. If a multipart upload task is canceled, the upload ID cannot be used to upload parts. In addition, the uploaded parts are deleted.

#include <alibabacloud/oss/OssClient.h>
using namespace AlibabaCloud::OSS;

int main(void)
{
    /* Initialize the information about the account used to access OSS. */
    std::string AccessKeyId = "yourAccessKeyId";
    std::string AccessKeySecret = "yourAccessKeySecret";
    std::string Endpoint = "yourEndpoint";
    /* Specify the name of the bucket. Example: examplebucket. */
    std::string BucketName = "examplebucket";
    /* Specify the full path of the object. Do not include the bucket name in the full path. Example: exampledir/exampleobject.txt.  */
    std::string ObjectName = "exampledir/exampleobject.txt";

    /* Initialize resources, such as network resources. */
    InitializeSdk();

    ClientConfiguration conf;
    OssClient client(Endpoint, AccessKeyId, AccessKeySecret, conf);
    InitiateMultipartUploadRequest initUploadRequest(BucketName, ObjectName);

    /* Initiate the multipart upload task. */
    auto uploadIdResult = client.InitiateMultipartUpload(initUploadRequest);
    auto uploadId = uploadIdResult.result().UploadId();

    /* Cancel the multipart upload task. */
    AbortMultipartUploadRequest  abortUploadRequest(BucketName, ObjectName, uploadId);
    auto abortUploadIdResult = client.AbortMultipartUpload(abortUploadRequest);

    if (!abortUploadIdResult.isSuccess()) {
        /* Handle exceptions. */
        std::cout << "AbortMultipartUpload fail" <<
        ",code:" << outcome.error().Code() <<
        ",message:" << outcome.error().Message() <<
        ",requestId:" << outcome.error().RequestId() << std::endl;
        return -1;
    }

    /* Release resources, such as network resources. */
    ShutdownSdk();
    return 0;
}

References

• For the complete sample code for multipart upload, visit GitHub.
• The following API operations are required to perform multipart upload:
  • The API operation that you can call to initiate a multipart upload task. For more information, see InitiateMultipartUpload.
  • The API operation that you can call to upload data by part. For more information, see UploadPart.
  • The API operation that you can call to complete a multipart upload task. For more information, see CompleteMultipartUpload.
• For more information about the API operation that you can call to cancel a multipart upload task, see AbortMultipartUpload.
• For more information about the API operation that you can call to list the uploaded parts, see ListParts.
• For more information about the API operation that you can call to list all ongoing multipart upload tasks, see ListMultipartUploads. Ongoing multipart upload tasks include tasks that have been initiated but are not completed and tasks that are canceled.