Object Storage Service (OSS) provides the multipart upload feature. Multipart upload allows you to split a large object into multiple parts to upload. After these parts are uploaded, you can call CompleteMultipartUpload to combine the parts into a complete object to implement resumable upload.

Background information

You can use the MultipartUpload method to upload a large object. In multipart upload, an object is split into multiple parts. Then, the parts are separately uploaded. If some parts fail to be uploaded, OSS records the upload progress. When you resume the upload, you need to upload only the parts that failed to be uploaded.

Notice To upload an object whose size is larger than 100 MB, we recommend that you use multipart upload to increase the upload success rate. If multipart upload is used to upload an object whose size is smaller than 100 MB and the value of the partSize parameter is inappropriate, the upload progress may not be fully displayed. To upload an object whose size is smaller than 100 MB, we recommend that you use simple upload.

If the ConnectionTimeoutError error occurs when you use the MultipartUpload method, you must fix the error. You can reduce the size of each part, extend the timeout period, or resend the request. You can also capture the ConnectionTimeoutError error and fix the error based on the cause. For more information, see Network connection timeout handling.

The following table describes the parameters that you can configure when you upload objects by using multipart upload.

Type Parameter Description
Required parameters name {String} The full path of the object. The path cannot contain the bucket name.
file {String|File} The path of the local file or HTML5 file.
[options] {Object} Optional parameters [checkpoint] {Object} The checkpoint file that records the execution result of the multipart upload task. If you enable resumable upload, you must configure this parameter. The checkpoint file stores the upload progress. If a part fails to be uploaded, OSS can resume the upload of the part based on the progress information recorded in this file. After the local file is uploaded, the file that stores progress information is deleted.
[parallel] {Number} The number of parts that can be uploaded in parallel. Default value: 5. You can use the default value if you do not have special requirements.
[partSize] {Number} The size of each part that you want to upload. Valid values: 100 KB to 5 GB. Default value: 1 MB (1 x 1024 x 1024). You can use the default value if you do not have special requirements.
[progress] {Function} The callback function that is used to query the progress of the upload task. The callback function can be an async function. The callback function includes the following parameters:
  • percentage {Number}: the progress of the upload task in percentage. Valid values: decimals between 0 and 1.
  • checkpoint {Object}: the checkpoint file that records the result of the multipart upload task.
  • res {Object}: the response that is returned when a part is uploaded.
[meta] {Object} The user metadata header. The header is prefixed with x-oss-meta-.
[mime] {String} The Content-Type request header.
[headers] {Object} Other headers. For more information, see RFC 2616. Examples:
  • Cache-Control: specifies the caching behavior by using specific commands in HTTP requests or responses. Example: Cache-Control: public, no-cache.
  • Content-Disposition: specifies whether the returned content is displayed as a web page or downloaded as an attachment and stored on your local device. Example: Content-Disposition: somename.
  • Content-Encoding: specifies the method that is used to compress the data of the specified media type. Example: Content-Encoding: gzip.
  • Expires: The expiration time of the cache. Unit: milliseconds.

Multipart upload examples

Notice OSS SDK for Node.js does not support MD5 verification in multipart upload tasks. We recommend that you call the CRC64 library to determine whether to perform CRC64 checks after the multipart upload task is completed.

The following code provides an example on how to use the multipartUpload method to perform multipart upload:

const OSS = require('ali-oss');
const path = require("path");

const client = new OSS({
  // Specify the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the region to oss-cn-hangzhou. 
  region: 'yourRegion',
  // The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. Using these credentials to perform operations in OSS is a high-risk operation. We recommend that you use a RAM user to call API operations or perform routine O&M. To create a RAM user, log on to the RAM console. 
  accessKeyId: 'yourAccessKeyId',
  accessKeySecret: 'yourAccessKeySecret',
  // Specify the name of the bucket. Example: examplebucket. 
  bucket: 'examplebucket',
});


const progress = (p, _checkpoint) => {
  // Record the upload progress of the object. 
  console.log(p); 
  // Record the checkpoint information about the multipart upload task. 
  console.log(_checkpoint); 
};

const headers = {
  // Specify the caching behavior of the web page when the object is downloaded. 
  'Cache-Control': 'no-cache',
  // Specify the name of the object when the object is downloaded. 
  'Content-Disposition': 'example.txt',
  // Specify the content encoding format of the object when the object is downloaded. 
  'Content-Encoding': 'utf-8', 
  // Specify the validity period of the request. Unit: milliseconds. 
  'Expires': '1000',
  // Specify the storage class of the object. 
  'x-oss-storage-class': 'Standard', 
  // Specify tags for the object. You can specify multiple tags for the object at the same time. 
  'x-oss-tagging': 'Tag1=1&Tag2=2', 
  // Specify whether existing objects are overwritten by objects with the same names when the multipart upload task is initialized. In this example, this parameter is set to true, which indicates that existing objects cannot be overwritten by objects with the same names. 
  'x-oss-forbid-overwrite': 'true'
}

// Start the multipart upload task. 
async function multipartUpload() {
  try {
    // Specify the full path of the object. Example: exampledir/exampleobject.txt. Then, specify the full path of the local file. Example: D:\\localpath\\examplefile.txt. The full path of the object cannot contain the bucket name. 
    // By default, if you set this parameter to the name of a local file, such as examplefile.txt, and you do not specify the local path, the local file is uploaded from the local path of the project to which the sample program belongs. 
    const result = await client.multipartUpload('exampledir/exampleobject.txt', path.normalize('D:\\localpath\\examplefile.txt'), {
      progress,
      // headers,
      // Configure the meta parameter to specify the metadata for the object. You can call the HeadObject operation to obtain the object metadata. 
      meta: {
        year: 2020,
        people: 'test',
      },
    });
    console.log(result);
    // Specify the full path of the object. Example: exampledir/exampleobject.txt. The full path of the object cannot contain the bucket name. 
    const head = await client.head('exampledir/exampleobject.txt');
    console.log(head);
  } catch (e) {
    // Handle timeout exceptions. 
    if (e.code === 'ConnectionTimeoutError') {
      console.log('TimeoutError');
      // do ConnectionTimeoutError operation
    }
    console.log(e);
  }
}

multipartUpload();

The multipartUpload method that is called in the preceding code contains the initMultipartUpload, uploadPart, and completeMultipartUpload operations. If you want to perform multipart upload step by step, call the preceding operations in sequence .initMultipartUpload, .uploadPart and .completeMultipartUpload .

Cancel a multipart upload task

You can use the client.abortMultipartUpload method to cancel a multipart upload task. If you cancel a multipart upload task, you cannot use the upload ID to upload parts. The uploaded parts are deleted.

The following code provides an example on how to cancel a multipart upload task:

const OSS = require('ali-oss');

const client = new OSS({
  // Specify the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the region to oss-cn-hangzhou. 
  region: 'yourRegion',
  // The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. Using these credentials to perform operations in OSS is a high-risk operation. We recommend that you use a RAM user to call API operations or perform routine O&M. To create a RAM user, log on to the RAM console. 
  accessKeyId: 'yourAccessKeyId',
  accessKeySecret: 'yourAccessKeySecret',
  // Specify the name of the bucket. Example: examplebucket. 
  bucket: 'examplebucket',
});

async function abortMultipartUpload() {
  // Specify the full path of the object. Example: exampledir/exampleobject.txt. The full path of the object cannot contain the bucket name. 
  const name = 'exampledir/exampleobject.txt';
  // Specify the upload ID. You can obtain the upload ID from the information that is returned by the multipartUpload method. 
  const uploadId = '0004B999EF518A1FE585B0C9360D****'; 
  const result = await client.abortMultipartUpload(name, uploadId);
  console.log(result);
}

abortMultipartUpload();

List multipart upload tasks

You can use the client.listUploads method to list all ongoing multipart upload tasks, which include tasks that have been initiated but are not completed and tasks that are canceled.

const OSS = require('ali-oss');

const client = new OSS({
  // Specify the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the region to oss-cn-hangzhou. 
  region: 'yourRegion',
  // The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. Using these credentials to perform operations in OSS is a high-risk operation. We recommend that you use a RAM user to call API operations or perform routine O&M. To create a RAM user, log on to the RAM console. 
  accessKeyId: 'yourAccessKeyId',
  accessKeySecret: 'yourAccessKeySecret',
  // Specify the name of the bucket. Example: examplebucket. 
  bucket: 'examplebucket',
});

async function listUploads(query = {}) {
  // You can configure parameters such as prefix, marker, delimiter, upload-id-marker, and max-uploads for query. 
  const result = await client.listUploads(query);

  result.uploads.forEach(upload => {
    // Specify the upload IDs of the multipart upload tasks. 
    console.log(upload.uploadId);
    // Combine all parts into a complete object and specify the full path of the object.  
    console.log(upload.name); 
  });
}

const query = {
  // Specify the maximum number of multipart upload tasks to return for the current list operation. The default value and the maximum value of max-uploads are 1000. 
  'max-uploads': 1000, 
};
listUploads(query);

List uploaded parts

You can use the client.listParts method to list all parts that are uploaded by using the specified upload ID in the multipart upload task.

  • List uploaded parts by using simple list

    The following code provides an example on how to list uploaded parts by using simple list:

    const OSS = require('ali-oss');
    
    const client = new OSS({
      // Specify the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the region to oss-cn-hangzhou. 
      region: 'yourRegion',
      // The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. Using these credentials to perform operations in OSS is a high-risk operation. We recommend that you use a RAM user to call API operations or perform routine O&M. To create a RAM user, log on to the RAM console. 
      accessKeyId: 'yourAccessKeyId',
      accessKeySecret: 'yourAccessKeySecret',
      // Specify the name of the bucket. Example: examplebucket. 
      bucket: 'examplebucket',
    });
    
    async function listParts() {
      const query = {
        // Specify the maximum number of parts to return for the current list operation. The default value and the maximum value of max-parts are both 1000. 
        'max-parts': 1000, 
      };
      // Specify the full path of the object. Example: exampledir/exampleobject.txt. Then, specify the upload ID. The full path of the object cannot contain the bucket name. 
      const result = await client.listParts('exampledir/exampleobject.txt', '0004B999EF518A1FE585B0C9360D****', query);
    
      result.parts.forEach(part => {
        console.log(part.PartNumber);
        console.log(part.LastModified);
        console.log(part.ETag);
        console.log(part.Size);
      });
    }
    
    listParts();
  • List all uploaded parts

    By default, you can list up to 1,000 parts at the same time by using client.listParts. If you want to list more than 1,000 uploaded parts, use the following code:

    const OSS = require('ali-oss');
    
    const client = new OSS({
      // Specify the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the region to oss-cn-hangzhou. 
      region: 'yourRegion',
      // The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. Using these credentials to perform operations in OSS is a high-risk operation. We recommend that you use a RAM user to call API operations or perform routine O&M. To create a RAM user, log on to the RAM console. 
      accessKeyId: 'yourAccessKeyId',
      accessKeySecret: 'yourAccessKeySecret',
      // Specify the name of the bucket. Example: examplebucket. 
      bucket: 'examplebucket',
    });
    
    async function listParts() {
      const query = {
        // Specify the maximum number of parts to return for the current list operation. The default value and the maximum value of max-parts are both 1000. 
        'max-parts': 1000, 
      };
      let result;
      do {
        // Specify the full path of the object. Example: exampledir/exampleobject.txt. Then, specify the upload ID. The full path of the object cannot contain the bucket name. 
        result = await client.listParts('exampledir/exampleobject.txt', '0004B999EF518A1FE585B0C9360D****', query);
        // Specify the position from which the list operation starts. Example: 2. Only the parts whose part numbers are greater than the value of this parameter are listed. 
        query['2'] = result.nextPartNumberMarker;
        result.parts.forEach(part => {
          console.log(part.PartNumber);
          console.log(part.LastModified);
          console.log(part.ETag);
          console.log(part.Size);
        });
      } while (result.isTruncated === "true");
    }
    
    listParts();

References

  • For more information about the complete sample code for multipart upload, visit GitHub.
  • The multipartUpload method that is used by OSS SDK for Node.js to perform multipart upload contains the following three API operations:
    • The API operation that you can call to initiate a multipart upload task. For more information, see InitiateMultipartUpload.
    • The API operation that you can call to upload data by part. For more information, see UploadPart.
    • The API operation that you can call to complete a multipart upload task. For more information, see CompleteMultipartUpload.
  • For more information about the API operation that you can call to cancel the multipart upload task, see AbortMultipartUpload.
  • For more information about the API operation that you can call to list the uploaded parts, see ListParts.
  • For more information about the API operation that you can call to list ongoing multipart upload tasks, see ListMultipartUploads. Ongoing multipart upload tasks include tasks that have been initiated but are not completed or are canceled.