Object Storage Service:How do I upload large objects to OSS?

Upload methods

You can upload objects that are 0 bytes to 48.8 TB in size to OSS. You can use alternative upload methods when you want to upload large objects or multiple objects at a time. These alternative methods are helpful in the following scenarios:

• The object that you want to upload is larger than 5 GB in size and cannot be uploaded by using the OSS console, simple upload, form upload, and append upload.

• The objects that you want to upload are smaller than 5 GB in size, but poor network performance frequently causes connection timeouts or dropped connections, and the uploads often fail.

Method 1: Multipart upload or resumable upload

Multipart upload and resumable upload are suitable for the following scenarios:

• Large objects: The objects that you want to upload are larger than 5 GB in size.

• Streaming upload: You want to start uploading objects in scenarios where the sizes of the objects are still uncertain, for example, in cloud video recording applications for video surveillance.

• Poor network connection: Resumable upload is recommended when you have poor network performance. Resumable upload allows you to upload only parts that have failed to be uploaded. This upload method is more efficient in scenarios where uploads are frequently interrupted because you do not need to re-upload parts that have been uploaded before the interruption.

• Upload acceleration: When the object that you want to upload is large, you can upload parts of the object in parallel to speed up the upload process.

For more information, see Multipart upload and Resumable upload.

Method 2: ossbrowser

Take note of the following items when you use ossbrowser to upload objects:

• You can use a Security Token Service (STS) token to log on to ossbrowser.

• To use the simple policy feature, you must log on to ossbrowser by using the AccessKey pair of a RAM user who has the RAM configuration permissions.

• When you upload a large number of objects, increase the number of concurrent upload tasks to speed up the upload.

• By default, ossbrowser uses multipart upload and resumable upload to upload objects. The size of the object that you want to upload cannot exceed 48.8 TB.

For more information, see ossbrowser 2.0.

Method 3: ossutil

Take note of the following items when you use ossutil to upload objects:

• You can use the cp command in ossutil to upload large objects.

• You can configure the --bigfile-threshold option to specify the size threshold above which objects are uploaded by using concurrent multipart upload.

• When you upload one or more large objects, you can configure the -jobs and -parallel options to control the number of concurrent tasks for optimal upload performance.

• If you want to upload a large number of objects, including large objects, to OSS, you can also use ossimport.

• If the number of objects that you want to upload at a time is small, we recommend that you use ossutil.

For more information, see Upload objects.

Troubleshooting

This section describes the common issues that you may encounter when you upload large objects and how to troubleshoot these issues:

• If you use PutObject to upload large objects, PutObject fails. This may be caused by the following factors:

  • The object that you want to upload is larger than 5 GB. Use ossutil to upload the object in parts.

  • The object that you want to upload is smaller than 5 GB. Due to poor network conditions, small objects are uploaded as expected, but when you upload a larger object, the request receives no response until a timeout error occurs. This may be because the maximum transmission unit (MTU) of the server is set to a large value. Run netstat -i on the ECS instance to query the current MTU setting. The following figure shows that the default MTU for the internal and external network interface cards is 1500.

    • Change the MTU to a smaller value. For example, you can run ip link set dev eth0 mtu 1470 on Linux to change the MTU for eth0 in the preceding figure to 1470.

    • Use the MultipartUpload operation or the ossutil tool.

• Can a single large object be uploaded concurrently?

  • If you use ossutil to upload the large object, you can specify the --bigfile-threshold, -jobs, and -parallel options to upload the object concurrently.

  • If you use the MultipartUpload operation of OSS APIs or OSS SDKs, write code to upload the object concurrently. Some OSS SDKs allow you to control the number of concurrent threads by using a parameter, such as taskNum in OSS SDK for Java or num_threads in OSS SDK for Python.