Object Storage Service:Multipart upload

Background information

You can use the MultipartUpload method to upload a large object. In multipart upload, an object is split into multiple parts. Then, the parts are separately uploaded. If some parts fail to be uploaded, OSS records the upload progress. When you resume the upload, you need to upload only the parts that failed to be uploaded.

If the ConnectionTimeoutError error occurs when you use the MultipartUpload method, you must fix the error. You can reduce the size of each part, extend the timeout period, or resend the request. You can also capture the ConnectionTimeoutError error and fix the error based on the cause. For more information, see Network connection timeout handling.

The following table describes the parameters that you can configure when you upload objects by using the multipart upload method.

Multipart upload examples

The following code provides an example on how to use the multipartUpload method to perform multipart upload:

const OSS = require('ali-oss');
const path = require("path");

const client = new OSS({
  // Specify the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the region to oss-cn-hangzhou. 
  region: 'yourRegion',
  // The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. Using these credentials to perform operations in OSS is a high-risk operation. We recommend that you use a RAM user to call API operations or perform routine O&M. To create a RAM user, log on to the RAM console. 
  accessKeyId: 'yourAccessKeyId',
  accessKeySecret: 'yourAccessKeySecret',
  // Specify the bucket name. Example: examplebucket. 
  bucket: 'examplebucket',
});


const progress = (p, _checkpoint) => {
  // Record the upload progress of the object. 
  console.log(p); 
  // Record the checkpoint information about the multipart upload task. 
  console.log(_checkpoint); 
};

const headers = {  
  // Specify the storage class of the object. 
  'x-oss-storage-class': 'Standard', 
  // Specify tags for the object. You can specify multiple tags for the object. 
  'x-oss-tagging': 'Tag1=1&Tag2=2', 
  // Specify whether existing objects are overwritten by objects with the same names when the multipart upload task is initialized. In this example, this parameter is set to true, which indicates that existing objects cannot be overwritten by objects with the same names. 
  'x-oss-forbid-overwrite': 'true'
}

// Start the multipart upload task. 
async function multipartUpload() {
  try {
    // Specify the full path of the object. Example: exampledir/exampleobject.txt. Then, specify the full path of the local file. Example: D:\\localpath\\examplefile.txt. The full path of the object cannot contain the bucket name. 
    // By default, if you set this parameter to the name of a local file, such as examplefile.txt, and you do not specify the local path, the local file is uploaded from the local path of the project to which the sample program belongs. 
    const result = await client.multipartUpload('exampledir/exampleobject.txt', path.normalize('D:\\localpath\\examplefile.txt'), {
      progress,
      // headers,
      // Configure the meta parameter to specify the metadata for the object. You can call the HeadObject operation to obtain the object metadata. 
      meta: {
        year: 2020,
        people: 'test',
      },
    });
    console.log(result);
    // Specify the full path of the object. Example: exampledir/exampleobject.txt. The full path of the object cannot contain the bucket name. 
    const head = await client.head('exampledir/exampleobject.txt');
    console.log(head);
  } catch (e) {
    // Handle timeout exceptions. 
    if (e.code === 'ConnectionTimeoutError') {
      console.log('TimeoutError');
      // do ConnectionTimeoutError operation
    }
    console.log(e);
  }
}

multipartUpload();

The multipartUpload method that is called in the preceding code contains the initMultipartUpload, uploadPart, and completeMultipartUpload operations. If you want to perform multipart upload step by step, call the preceding operations in sequence. For more information, see .initMultipartUpload, .uploadPart, and .completeMultipartUpload.

Cancel a multipart upload task

You can use the client.abortMultipartUpload method to cancel a multipart upload task. If you cancel a multipart upload task, you cannot use the upload ID to upload parts. The uploaded parts are deleted.

The following sample code provides an example on how to cancel a multipart upload task:

const OSS = require('ali-oss');

const client = new OSS({
  // Specify the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the region to oss-cn-hangzhou. 
  region: 'yourRegion',
  // The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. Using these credentials to perform operations in OSS is a high-risk operation. We recommend that you use a RAM user to call API operations or perform routine O&M. To create a RAM user, log on to the RAM console. 
  accessKeyId: 'yourAccessKeyId',
  accessKeySecret: 'yourAccessKeySecret',
  // Specify the bucket name. Example: examplebucket. 
  bucket: 'examplebucket',
});

async function abortMultipartUpload() {
  // Specify the full path of the object. Example: exampledir/exampleobject.txt. The full path of the object cannot contain the bucket name. 
  const name = 'exampledir/exampleobject.txt';
  // Specify the upload ID. You can obtain the upload ID from the information that is returned by the multipartUpload method. 
  const uploadId = '0004B999EF518A1FE585B0C9360D****'; 
  const result = await client.abortMultipartUpload(name, uploadId);
  console.log(result);
}

abortMultipartUpload();

List multipart upload tasks

You can call the client.listUploads method to list all ongoing multipart upload tasks, which are tasks that have been initiated but are not completed or canceled.

const OSS = require('ali-oss');

const client = new OSS({
  // Specify the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the region to oss-cn-hangzhou. 
  region: 'yourRegion',
  // The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. Using these credentials to perform operations in OSS is a high-risk operation. We recommend that you use a RAM user to call API operations or perform routine O&M. To create a RAM user, log on to the RAM console. 
  accessKeyId: 'yourAccessKeyId',
  accessKeySecret: 'yourAccessKeySecret',
  // Specify the bucket name. Example: examplebucket. 
  bucket: 'examplebucket',
});

async function listUploads(query = {}) {
  // You can configure the following parameters for query: prefix, marker, delimiter, upload-id-marker, and max-uploads. 
  const result = await client.listUploads(query);

  result.uploads.forEach(upload => {
    // Specify the upload IDs of the multipart upload tasks. 
    console.log(upload.uploadId);
    // Combine all parts into a complete object and specify the full path of the object.  
    console.log(upload.name); 
  });
}

const query = {
  // Specify the maximum number of multipart upload tasks to return for the current list operation. The default value and the maximum value of max-uploads are 1000. 
  'max-uploads': 1000, 
};
listUploads(query);

List uploaded parts

You can use the client.listParts method to list all parts that are uploaded by using the specified upload ID in the multipart upload task.

• List uploaded parts by using simple list

  The following code provides an example on how to list uploaded parts by using simple list:

  const OSS = require('ali-oss');
  
  const client = new OSS({
    // Specify the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the region to oss-cn-hangzhou. 
    region: 'yourRegion',
    // The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. Using these credentials to perform operations in OSS is a high-risk operation. We recommend that you use a RAM user to call API operations or perform routine O&M. To create a RAM user, log on to the RAM console. 
    accessKeyId: 'yourAccessKeyId',
    accessKeySecret: 'yourAccessKeySecret',
    // Specify the bucket name. Example: examplebucket. 
    bucket: 'examplebucket',
  });
  
  async function listParts() {
    const query = {
      // Specify the maximum number of parts to return for the current list operation. The default value and the maximum value of max-parts are both 1000. 
      'max-parts': 1000, 
    };
    // Specify the full path of the object. Example: exampledir/exampleobject.txt. Then, specify the upload ID. The full path of the object cannot contain the bucket name. 
    const result = await client.listParts('exampledir/exampleobject.txt', '0004B999EF518A1FE585B0C9360D****', query);
  
    result.parts.forEach(part => {
      console.log(part.PartNumber);
      console.log(part.LastModified);
      console.log(part.ETag);
      console.log(part.Size);
    });
  }
  
  listParts();

• List all uploaded parts

  By default, you can list up to 1,000 parts at the same time by using client.listParts. If you want to list more than 1,000 uploaded parts, use the following code:

  const OSS = require('ali-oss');
  
  const client = new OSS({
    // Specify the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the region to oss-cn-hangzhou. 
    region: 'yourRegion',
    // The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. Using these credentials to perform operations in OSS is a high-risk operation. We recommend that you use a RAM user to call API operations or perform routine O&M. To create a RAM user, log on to the RAM console. 
    accessKeyId: 'yourAccessKeyId',
    accessKeySecret: 'yourAccessKeySecret',
    // Specify the bucket name. Example: examplebucket. 
    bucket: 'examplebucket',
  });
  
  async function listParts() {
    const query = {
      // Specify the maximum number of parts to return for the current list operation. The default value and the maximum value of max-parts are both 1000. 
      'max-parts': 1000, 
    };
    let result;
    do {
      // Specify the full path of the object. Example: exampledir/exampleobject.txt. Then, specify the upload ID. The full path of the object cannot contain the bucket name. 
      result = await client.listParts('exampledir/exampleobject.txt', '0004B999EF518A1FE585B0C9360D****', query);
      // Specify the position from which the list operation starts. Example: 2. Only the parts whose part numbers are greater than the value of this parameter are listed. 
      query['2'] = result.nextPartNumberMarker;
      result.parts.forEach(part => {
        console.log(part.PartNumber);
        console.log(part.LastModified);
        console.log(part.ETag);
        console.log(part.Size);
      });
    } while (result.isTruncated === "true");
  }
  
  listParts();

FAQ

How do I obtain the MD5 hash of the object after I upload the object by using multipart upload?

When you use the CompleteMultipartUpload method, the value of the Content-MD5 header in the response is the MD5 hash of the object.

References

• For more information about the complete sample code for multipart upload, visit GitHub.
• The multipartUpload method that is used by OSS SDK for Node.js to perform multipart upload contains the following three API operations:
  • The API operation that you can call to initiate a multipart upload task. For more information, see InitiateMultipartUpload.
  • The API operation that you can call to upload data by part. For more information, see UploadPart.
  • The API operation that you can call to complete a multipart upload task. For more information, see CompleteMultipartUpload.
• For more information about the API operation that you can call to cancel a multipart upload task, see AbortMultipartUpload.
• For more information about the API operation that you can call to list the uploaded parts, see ListParts.
• For more information about the API operation that you can call to list ongoing multipart upload tasks, see ListMultipartUploads. Ongoing multipart upload tasks include tasks that have been initiated but are not completed and or canceled.