Throttle uploads and downloads of files by using Python SDK - Object Storage Service

This topic describes how to add parameters in an object upload or download request to set the limit of upload or download bandwidth. This ensures sufficient bandwidth for other applications.

Usage notes

In this topic, the public endpoint of the China (Hangzhou) region is used. If you want to access OSS from other Alibaba Cloud services in the same region as OSS, use an internal endpoint. For more information about OSS regions and endpoints, see Regions and endpoints.
In this topic, access credentials are obtained from environment variables. For more information about how to configure access credentials, see Configure access credentials using OSS SDK for Python 1.0.
In this topic, an OSSClient instance is created by using an OSS endpoint. If you want to create an OSSClient instance by using custom domain names or Security Token Service (STS), see Initialization.

Configure single-connection bandwidth throttling for simple upload and download

The following sample code provides an example on how to configure single-connection bandwidth throttling for simple upload and download:

# -*- coding: utf-8 -*-

import oss2
from oss2.credentials import EnvironmentVariableCredentialsProvider
from oss2.models import OSS_TRAFFIC_LIMIT

# Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured.
auth = oss2.ProviderAuthV4(EnvironmentVariableCredentialsProvider())

# Specify the endpoint of the region where the bucket is located. For example, if the bucket is in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com.
endpoint = "https://oss-cn-hangzhou.aliyuncs.com"
# Specify the region that corresponds to the endpoint, for example, cn-hangzhou. Note: This parameter is required for V4 signatures.
region = "cn-hangzhou"

# Set the bucket name to examplebucket.
bucket = oss2.Bucket(auth, endpoint, "examplebucket", region=region)

# Specify the full path of the object. The full path cannot contain the bucket name. Example: exampledir/exampleobject.txt.
object_name = 'exampledir/exampleobject.txt'
# Specify the full path of the local file to upload. Example: D:\\localpath\\examplefile.txt. If a local path is not specified, the file is uploaded from the directory of the sample program.
local_file_name = 'D:\\localpath\\examplefile.txt'
# Specify the full path to which the object is downloaded. If the specified local file exists, it is overwritten. If the file does not exist, it is created.
# If a local path is not specified, the downloaded file is saved to the directory of the sample program.
down_file_name = 'D:\\localpath\\exampleobject.txt'

# Set the throttling speed to 100 KB/s (819,200 bit/s) in the headers.
limit_speed = (100 * 1024 * 8)
headers = dict()
headers[OSS_TRAFFIC_LIMIT] = str(limit_speed)

# Throttle the file upload.
result = bucket.put_object_from_file(object_name, local_file_name, headers=headers)
print('http response status:', result.status)

# Throttle the download of the file to your local machine.
result = bucket.get_object_to_file(object_name, down_file_name, headers=headers)
print('http response status:', result.status)

Throttle multipart uploads

The following code shows how to throttle multipart uploads:

# -*- coding: utf-8 -*-
import os
from oss2 import SizedFileAdapter, determine_part_size
from oss2.headers import OSS_TRAFFIC_LIMIT
from oss2.models import PartInfo
import oss2
from oss2.credentials import EnvironmentVariableCredentialsProvider

# Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured.
auth = oss2.ProviderAuthV4(EnvironmentVariableCredentialsProvider())

# Specify the endpoint of the region where the bucket is located. For example, if the bucket is in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com.
endpoint = "https://oss-cn-hangzhou.aliyuncs.com"
# Specify the region that corresponds to the endpoint, for example, cn-hangzhou. Note: This parameter is required for V4 signatures.
region = "cn-hangzhou"

# Set the bucket name to examplebucket.
bucket = oss2.Bucket(auth, endpoint, "examplebucket", region=region)

# Specify the full path of the object. The full path cannot contain the bucket name. Example: exampledir/exampleobject.txt.
key = 'exampledir/exampleobject.txt'
# Specify the full path of the local file. Example: D:\\localpath\\examplefile.txt.
filename = 'D:\\localpath\\examplefile.txt'

total_size = os.path.getsize(filename)
# The determine_part_size method is used to determine the part size.
part_size = determine_part_size(total_size, preferred_size=100 * 1024)

# Initialize the multipart upload.
upload_id = bucket.init_multipart_upload(key).upload_id
parts = []

# Set the throttling speed to 100 KB/s (819,200 bit/s) in the headers.
limit_speed = (100 * 1024 * 8)
headers = dict()
headers[OSS_TRAFFIC_LIMIT] = str(limit_speed)

# Upload parts one by one.
with open(filename, 'rb') as fileobj:
    part_number = 1
    offset = 0
    while offset < total_size:
        num_to_upload = min(part_size, total_size - offset)
        # Use SizedFileAdapter to wrap the file object so that only the data for the current part is uploaded.
        result = bucket.upload_part(key, upload_id, part_number,
                                    SizedFileAdapter(fileobj, num_to_upload), headers=headers)
        parts.append(PartInfo(part_number, result.etag))

        offset += num_to_upload
        part_number += 1

# Complete the multipart upload.
# To set headers when you complete the multipart upload, see the following sample code.
headers = dict()
# Set the access control list (ACL) for the object. In this example, the ACL is set to OBJECT_ACL_PRIVATE, which specifies the private permission.
# headers["x-oss-object-acl"] = oss2.OBJECT_ACL_PRIVATE
bucket.complete_multipart_upload(key, upload_id, parts, headers=headers)
# bucket.complete_multipart_upload(key, upload_id, parts)

# Verify the multipart upload.
with open(filename, 'rb') as fileobj:
    assert bucket.get_object(key).read() == fileobj.read()

Configure bandwidth throttling for upload and download that use signed URLs

The following code shows how to throttle file uploads and downloads that use signed URLs:

# -*- coding: utf-8 -*-

import oss2
from oss2.credentials import EnvironmentVariableCredentialsProvider
from oss2.models import OSS_TRAFFIC_LIMIT

# Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured.
auth = oss2.ProviderAuthV4(EnvironmentVariableCredentialsProvider())

# Specify the endpoint of the region where the bucket is located. For example, if the bucket is in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com.
endpoint = "https://oss-cn-hangzhou.aliyuncs.com"
# Specify the region that corresponds to the endpoint, for example, cn-hangzhou. Note: This parameter is required for V4 signatures.
region = "cn-hangzhou"

# Set the bucket name to examplebucket.
bucket = oss2.Bucket(auth, endpoint, "examplebucket", region=region)

# Specify the full path of the object. The full path cannot contain the bucket name. Example: exampledir/exampleobject.txt.
object_name = 'exampledir/exampleobject.txt'
# Specify the full path of the local file to upload. Example: D:\\localpath\\examplefile.txt. If a local path is not specified, the file is uploaded from the directory of the sample program.
local_file_name = 'D:\\localpath\\examplefile.txt'
# Specify the full path to which the object is downloaded. If the specified local file exists, it is overwritten. If the file does not exist, it is created.
# If a local path is not specified, the downloaded file is saved to the directory of the sample program.
down_file_name = 'D:\\localpath\\exampleobject.txt'

# Set the throttling speed to 100 KB/s (819,200 bit/s) in the params.
limit_speed = (100 * 1024 * 8)
params = dict()
params[OSS_TRAFFIC_LIMIT] = str(limit_speed)

# Create a signed URL for the throttled upload. The URL is valid for 60 seconds.
url = bucket.sign_url('PUT', object_name, 60, params=params)
print('put object url:', url)

# Throttle the upload.
result = bucket.put_object_with_url_from_file(url, local_file_name)
print('http response status:', result.status)

# Create a signed URL for the throttled download. The URL is valid for 60 seconds.
url = bucket.sign_url('GET', object_name, 60, params=params)
print('get object url:', url)

# Throttle the download.
result = bucket.get_object_with_url_to_file(url, down_file_name)
print('http response status:', result.status)

References

For the complete sample code for single-connection bandwidth throttling, see GitHub example.