By using multipart upload and resumable upload provided by Alibaba Cloud OSS, you can split an object into multiple data blocks (parts) and upload them separately. After uploading all the parts, you can call an API operation to combine them into a complete object.

Scenarios

When you use simple upload (by calling the PutObject operation) to upload a large object to OSS, the upload may fail due to a network error. In this case, you must upload the entire object again, which is a waste of resources. To resolve this issue, you can use multipart upload to resume upload from the last uploaded part.

Compared with other upload methods, multipart upload is applicable to the following scenarios:
  • Poor network connectivity: If a part fails to be uploaded by using a mobile phone, you can re-upload only the failed part instead of all parts.
  • Resumable upload required: An upload in progress can be paused and resumed at any time.
  • Upload acceleration: When the object to be uploaded to OSS is large, multiple parts can be uploaded concurrently to speed up the process.
  • Streaming upload: Objects of unknown sizes can be uploaded at any time. This scenario is common in industry applications such as video surveillance.

Implementation modes for multipart upload

Implementation mode Description
ossutil High-performance command-line tool
Java SDK SDK demos for various programming languages
Python SDK
PHP SDK
Go SDK
C SDK
.NET SDK

Implementation modes for resumable upload

If the system stops responding during a multipart upload, you can resume the upload by calling the ListMultipartUploads and ListParts operations to retrieve all multipart upload tasks on an object and list the uploaded parts in each task. This method allows an upload task to be resumed from the last uploaded part. The same principles apply if you pause and then resume an upload.
Implementation mode Description
Java SDK SDK demos for various programming languages
Python SDK
Go SDK
C SDK
.NET SDK
Android SDK
iOS SDK

Upload limits

  • Size: The maximum size of an object is determined by the size of parts. Multipart upload supports a maximum of 10,000 parts. Each part must be at least 100 KB and no more than 5 GB, with the exception of the last part. Therefore, the object size cannot exceed 48.8 TB.
  • Object names must comply with the following naming conventions:
    • Object names must be encoded in UTF-8.
    • Object names must be one byte to 1,023 bytes in length.
    • Object names cannot start with a forward slash (/) or a backslash (\).
  • If you upload a large number of objects with sequential prefixes such as timestamps and letters in the object names, multiple object indexes may be stored in a single partition. If too many requests are sent to query these objects, the responsiveness may be slow. We recommend that you do not upload a large number of objects with sequential prefixes. For more information about how to change sequential prefixes to random prefixes, see OSS performance and scalability best practices.
  • By default, when you upload an object to OSS, the existing object with the same object name will be overwritten. You can use the following methods to protect your objects from being unexpectedly overwritten:
    • Enable versioning

      If versioning is enabled, deleted or overwritten objects are saved as previous versions. You can recover the previous versions at any time. For more information, see Overview.

    • Include a parameter in the upload request header

      Include the x-oss-forbid-overwrite parameter in the upload request header, and set its value to true. This way, when you upload an object whose object name is the same as that of an existing object, the object cannot be uploaded. OSS returns the FileAlreadyExists error. For more information, see InitiateMultipartUpload.

      If this parameter is not included in the request header or the value of this parameter is false, the object with the same object name is overwritten.

Multipart upload process

The multipart upload process is as follows:

  1. Split the object to upload into multiple parts.
  2. Call the InitiateMultipartUpload operation to initiate a multipart upload task.
  3. Call the UploadPart operation to upload the parts one by one or concurrently.
  4. Call the CompleteMultipartUpload operation to complete the upload.

Before you start a multipart upload, you need to note that:

  • Each part cannot be smaller than 100 KB, with the exception of the last part. Otherwise, you may fail to call the CompleteMultipartUpload operation.
  • After the object to upload is split into parts, they are sorted by partNumber specified during the upload. However, upload in sequence is not required. Therefore, the parts can be uploaded concurrently.

    Due to network conditions and the device load, the upload does not necessarily speed up when more parts are uploaded concurrently. We recommend that you increase the part size when network conditions are good, and decrease the part size when network conditions are bad.

  • By default, when all parts are uploaded but you have not called the CompleteMultipartUpload operation, the uploaded parts are not deleted automatically. You can call the AbortMultipartUpload operation to terminate the upload and delete the parts that occupy the storage space. For more information about how to automatically delete the uploaded parts, see Manage object lifecycles.

Upload security and authorization

To prevent unauthorized third-party users from uploading data to your bucket, OSS provides bucket- and object-level access control. For more information, see Overview.

To authorize third-party users to upload objects, OSS also provides account authorization. For more information, see Authorized third-party upload.

Subsequent operations

  • After uploading objects to OSS, you can use Upload callback to submit a callback request to the specified application server and perform subsequent operations.
  • Images can be processed after they are uploaded. For more information, see Image processing.
  • Audio or video objects can be processed after they are uploaded. For more information, see Cloud data processing.

API reference