Object Storage Service:Multipart upload (Node.js SDK)

Background information

When you need to upload a large file, you can use the MultipartUpload method to perform a multipart upload. This method splits an object into multiple parts that are uploaded separately. If some parts fail to upload, OSS records the upload progress. When you re-upload, you only need to upload the failed parts instead of the entire file.

When you use the MultipartUpload method, you must handle ConnectionTimeoutError errors if they occur. For example, you can handle the timeout by reducing the shard size, increasing the timeout period, retrying the request, or catching the ConnectionTimeoutError. For more information, see Network error handling.

The following table describes the parameters related to multipart uploads.

Complete multipart upload example

The following code shows how to use the multipartUpload method to perform a multipart upload.

const OSS = require('ali-oss');
const path = require("path");

const client = new OSS({
  // Replace yourregion with the region where the bucket is located. For example, if the bucket is in the China (Hangzhou) region, set the region to oss-cn-hangzhou.
  region: 'yourregion',
  // Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are set.
  accessKeyId: process.env.OSS_ACCESS_KEY_ID,
  accessKeySecret: process.env.OSS_ACCESS_KEY_SECRET,
  authorizationV4: true,
  // Specify the bucket name.
  bucket: 'yourbucketname',
});


const progress = (p, _checkpoint) => {
  // The upload progress of the object.
  console.log(p); 
  // The breakpoint information for the multipart upload.
  console.log(_checkpoint); 
};

const headers = {  
  // Specify the storage class of the object.
  'x-oss-storage-class': 'Standard', 
  // Specify tags for the object. You can specify multiple tags.
  'x-oss-tagging': 'Tag1=1&Tag2=2', 
  // Specify whether to overwrite an object that has the same name when you initialize the multipart upload. In this example, this parameter is set to true to prevent overwriting an object that has the same name.
  'x-oss-forbid-overwrite': 'true'
}

// Start the multipart upload.
async function multipartUpload() {
  try {
    // Specify the full path of the object (for example, exampledir/exampleobject.txt) and the full path of the local file (for example, D:\\localpath\\examplefile.txt). The full path of the object cannot contain the bucket name.
    // If you do not specify a local path in the full path of the local file (for example, examplefile.txt), the file is uploaded from the local path that corresponds to the project where the sample program is located.
    const result = await client.multipartUpload('exampledir/exampleobject.txt', path.normalize('D:\\localpath\\examplefile.txt'), {
      progress,
      // headers,
      // Specify the meta parameter to customize the metadata of the object. You can use the head operation to obtain the metadata of the object.
      meta: {
        year: 2020,
        people: 'test',
      },
    });
    console.log(result);
    // Specify the full path of the object, for example, exampledir/exampleobject.txt. The full path of the object cannot contain the bucket name.
    const head = await client.head('exampledir/exampleobject.txt');
    console.log(head);
  } catch (e) {
    // Catch the timeout exception.
    if (e.code === 'ConnectionTimeoutError') {
      console.log('TimeoutError');
      // do ConnectionTimeoutError operation
    }
    console.log(e);
  }
}

multipartUpload();

The multipartUpload method used in the preceding example encapsulates three API operations: initializing a multipart upload, uploading parts, and completing a multipart upload. If you want to implement a multipart upload in steps, you can call the .initMultipartUpload, .uploadPart, and .completeMultipartUpload methods in sequence.

Cancel a multipart upload event

You can call the client.abortMultipartUpload method to cancel a multipart upload task. After a multipart upload task is canceled, you can no longer use the upload ID to upload parts, and the uploaded parts are deleted.

The following code shows how to cancel a multipart upload.

const OSS = require("ali-oss");

const client = new OSS({
  // Replace yourregion with the region where the bucket is located. For example, if the bucket is in the China (Hangzhou) region, set the region to oss-cn-hangzhou.
  region: "yourregion",
  // Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are set.
  accessKeyId: process.env.OSS_ACCESS_KEY_ID,
  accessKeySecret: process.env.OSS_ACCESS_KEY_SECRET,
  authorizationV4: true,
  // Specify the bucket name.
  bucket: "yourbucketname",
});

async function abortMultipartUpload() {
  // Specify the full path of the object, for example, exampledir/exampleobject.txt. The full path of the object cannot contain the bucket name.
  const name = "exampledir/exampleobject.txt";
  // Specify the upload ID. The upload ID is obtained from the response to the call of InitiateMultipartUpload to initialize the multipart upload.
  const uploadId = "0004B999EF518A1FE585B0C9360D****";
  const result = await client.abortMultipartUpload(name, uploadId);
  console.log(result);
}

abortMultipartUpload();

List multipart upload events

You can call the client.listUploads method to list all in-progress multipart uploads. In-progress multipart uploads are uploads that have been initialized but are not yet completed or canceled.

const OSS = require("ali-oss");

const client = new OSS({
  // Replace yourregion with the region where the bucket is located. For example, if the bucket is in the China (Hangzhou) region, set the region to oss-cn-hangzhou.
  region: "yourregion",
  // Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are set.
  accessKeyId: process.env.OSS_ACCESS_KEY_ID,
  accessKeySecret: process.env.OSS_ACCESS_KEY_SECRET,
  authorizationV4: true,
  // Specify the bucket name.
  bucket: "yourbucketname",
});

async function listUploads(query = {}) {
  // You can set parameters such as prefix, marker, delimiter, upload-id-marker, and max-uploads in the query.
  const result = await client.listUploads(query);

  result.uploads.forEach((upload) => {
    // The upload ID of the multipart upload.
    console.log(upload.uploadId);
    // Combine all uploaded parts into a complete object and specify the full path of the object.
    console.log(upload.name);
  });
}

const query = {
  // Specify the maximum number of multipart upload events to return. The default and maximum value of the max-uploads parameter is 1000.
  "max-uploads": 1000,
};
listUploads(query);

List uploaded parts

During a multipart upload, you can call the client.listParts method to list all successfully uploaded parts for a specified upload ID.

const OSS = require("ali-oss");

const client = new OSS({
  // Replace yourregion with the region where the bucket is located. For example, if the bucket is in the China (Hangzhou) region, set the region to oss-cn-hangzhou.
  region: "yourregion",
  // Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are set.
  accessKeyId: process.env.OSS_ACCESS_KEY_ID,
  accessKeySecret: process.env.OSS_ACCESS_KEY_SECRET,
  authorizationV4: true,
  // Specify the bucket name.
  bucket: "yourbucketname",
});

async function listParts() {
  const query = {
    // Specify the maximum number of parts to return. The default and maximum value of the max-parts parameter is 1000.
    "max-parts": 1000,
  };
  let result;
  do { 
    result = await client.listParts(
      // Specify the full path of the object (for example, exampledir/exampleobject.txt). The full path of the object cannot contain the bucket name.
      "exampledir/exampleobject.txt",
      // The upload ID is obtained from the response after you call InitiateMultipartUpload to initialize the multipart upload and before you call CompleteMultipartUpload to complete the multipart upload.
      "0004B999EF518A1FE585B0C9360D****",
      query
    );
    // Specify the starting position for the next list operation. Only parts with part numbers greater than this value are listed.
    query["part-number-marker"] = result.nextPartNumberMarker;
    result.parts.forEach((part) => {
      console.log(part.PartNumber);
      console.log(part.LastModified);
      console.log(part.ETag);
      console.log(part.Size);
    });
  } while (result.isTruncated === "true");
}

listParts();

References

• For the complete sample code for multipart upload, see GitHub examples.

• The multipartUpload method in the Node.js SDK encapsulates the following three API operations:

  • For more information about the API operation to initialize a multipart upload, see InitiateMultipartUpload.

  • For more information about the API operation to upload a part, see UploadPart.

  • For more information about the API operation to complete a multipart upload, see CompleteMultipartUpload.

• For more information about the API operation to cancel a multipart upload, see AbortMultipartUpload.

• For more information about the API operation to list uploaded parts, see ListParts.

• For more information about the API operation to list all in-progress multipart uploads (uploads that are initialized but not yet completed or canceled), see ListMultipartUploads.