Data can be uploaded to Object Storage Service (OSS) directly from clients. This solution accelerates uploads and reduces the resource usage of the application server by eliminating the need to transfer objects to and from the application server. This topic describes the benefits, implementation methods, and best practices of this solution.
Benefits
In the traditional server and client architecture, uploading data to OSS from the application server is the common upload method. The client uploads objects to the application server, and then the application server uploads the objects to OSS. In the preceding process, data must be transferred twice over the network. This results in a waste of network resources and an increase in the resource usage of the application server. To resolve this issue, you can upload objects directly to OSS without the need to upload objects to the application server.
Implementation methods
To upload data to OSS from clients, you must resolve the following issues:
CORS
If your client is a web client or mini program, you need to solve the problem of cross-origin resource sharing (CORS) being prohibited. To ensure data security, browsers and mini programs usually prohibit CORS. This also prevents your client code from being used to connect to OSS. You can configure CORS rules for an OSS bucket to allow the domain name of your web client or mini program to access OSS. For more information about how to configure CORS rules, see CORS.
Security authorization
To upload an object to OSS, you must use the AccessKey pair of a RAM user to complete signature authentication. However, if the client uses a permanent AccessKey pair, the AccessKey pair may be leaked. This may cause security risks. To resolve this issue, you can use one of the following solutions to implement secure upload:
Obtain temporary access credentials from STS on the application server
In most object upload scenarios, we recommend that you use Security Token Service (STS) SDKs to obtain temporary access credentials on the application server, and then use the temporary access credentials and OSS SDKs to upload objects from the client directly to OSS. The client can reuse the temporary access credentials on the application server to generate a signature. This is suitable for scenarios in which multipart upload and resumable upload are used to upload large objects. However, frequent calls to STS may cause throttling. In this case, we recommend that you cache the temporary access credentials and renew them before the validity period ends. For more information, see What is STS?
Obtain a signature and policy from the application server
In scenarios in which you want to limit the attributes of an object that you want to upload, you can obtain information, such as a signature and policy that are required to call the PostObject operation, from the application server. Then, the client can use the obtained information to upload the object without using OSS SDKs. You can use the policy generated by the server to limit the attributes of an object that you want to upload, such as the object size and type. This solution is suitable for uploading objects by using HTML forms. This solution does not support multipart upload and resumable upload. For more information, see PostObject.
Obtain a signed URL from the application server
For simple object upload scenarios, you can use OSS SDKs on the application server to obtain a signed URL that is required to call the PutObject operation. Then, the client can use the signed URL to upload an object without using OSS SDKs. This solution is not suitable for multipart upload and resumable upload. The application server generates a signed URL for each part and returns the signed URL to the client. This increases the number of interactions with the application server and the complexity of network requests. In addition, the client may modify the content or order of the parts. This results in an invalid combined object. For more information, see Add signatures to URLs.
Obtain temporary access credentials from STS on the application server
The following figure shows how to upload an object to OSS from a client by using temporary access credentials from STS obtained on the application server.
The client requests temporary access credentials from the application server.
The application server uses STS SDKs to call the AssumeRole operation to obtain temporary access credentials.
STS generates and returns the temporary access credentials to the application server.
The application server returns the temporary access credentials to the client.
The client uses OSS SDKs to upload an object to OSS by using the temporary access credentials.
The following sample code provides an example on how to obtain temporary access credentials from STS on the application server:
import json
from alibabacloud_tea_openapi.models import Config
from alibabacloud_sts20150401.client import Client as Sts20150401Client
from alibabacloud_sts20150401 import models as sts_20150401_models
from alibabacloud_credentials.client import Client as CredentialClient
# Replace <your_role_arn> with the Alibaba Cloud Resource Name (ARN) of the RAM user that has the permissions to upload objects to the specified OSS bucket.
role_arn_for_oss_upload = '<your_role_arn>'
# Replace <your_region_id> with the ID of the region of the STS service. Example: cn-hangzhou.
region_id = '<your_region_id>'
def get_sts_token():
# If you do not specify parameters when you initialize CredentialClient, the default credential chain is used.
# If you run the program on your local computer, you can specify the AccessKey pair by configuring the ALIBABA_CLOUD_ACCESS_KEY_ID and ALIBABA_CLOUD_ACCESS_KEY_SECRET environment variables.
# If you run the program on an Elastic Compute Service (ECS) instance, an elastic container instance, or a node in a Container Service for Kubernetes (ACK) cluster, you can specify the bound instance or node by configuring the ALIBABA_CLOUD_ECS_METADATA environment variable. The SDK automatically exchanges the temporary access credentials obtained from STS.
config = Config(region_id=region_id, credential=CredentialClient())
sts_client = Sts20150401Client(config=config)
assume_role_request = sts_20150401_models.AssumeRoleRequest(
role_arn=role_arn_for_oss_upload,
# Replace <your_role_session_name> with a custom session name, such as oss-role-session.
role_session_name='<your_role_session_name>'
)
response = sts_client.assume_role(assume_role_request)
token = json.dumps(response.body.credentials.to_map())
return token
Sample code for the client
The following sample code provides an example on how to use the temporary access credentials to upload an object to OSS from a web client:
let credentials = null;
const form = document.querySelector("form");
form.addEventListener("submit", async (event) => {
event.preventDefault();
// Re-obtain the temporary access credentials when they expire. This reduces the number of STS calls.
if (isCredentialsExpired(credentials)) {
const response = await fetch("/get_sts_token_for_oss_upload", {
method: "GET",
});
credentials = await response.json();
}
const client = new OSS({
// Replace <your_bucket> with the name of the OSS bucket.
bucket: "<your_bucket>",
// Replace <your_region> with the ID of the region in which the OSS bucket is located. Example: oss-cn-hangzhou.
region: "oss-<your_region>",
accessKeyId: credentials.AccessKeyId,
accessKeySecret: credentials.AccessKeySecret,
stsToken: credentials.SecurityToken,
});
const fileInput = document.querySelector("#file");
const file = fileInput.files[0];
const result = await client.put(file.name, file);
console.log(result);
});
/**
* Determine whether the temporary access credentials expire.
**/
function isCredentialsExpired(credentials) {
if (!credentials) {
return true;
}
const expireDate = new Date(credentials.Expiration);
const now = new Date();
// If the remaining validity period of the temporary access credentials is less than 1 minute, the temporary access credentials are considered expired.
return expireDate.getTime() - now.getTime() <= 60000;
}
Obtain a signature and policy from the application server
The following figure shows how to upload an object from a client to OSS by using the signature and policy obtained from the application server.
The client requests information, such as a signature and policy, from the application server.
The application server generates and returns information, such as the signature and policy, to the client.
The client uses information, such as the signature and policy, to call the PostObject operation to upload the object to OSS by using an HTML form.
The following sample code provides an example on how to obtain a signature and policy from the application server:
import os
from hashlib import sha1 as sha
import json
import base64
import hmac
import datetime
import time
# Configure the OSS_ACCESS_KEY_ID environment variable.
access_key_id = os.environ.get('OSS_ACCESS_KEY_ID')
# Configure the OSS_ACCESS_KEY_SECRET environment variable.
access_key_secret = os.environ.get('OSS_ACCESS_KEY_SECRET')
# Replace <your-bucket> with the name of the bucket.
bucket = '<your-bucket>'
# Set host to a value that is in the bucketname.endpoint format. Replace <your-bucket> with the name of the bucket. Replace <your-endpoint> with the OSS endpoint. Example: oss-cn-hangzhou.aliyuncs.com.
host = 'https://<your-bucket>.<your-endpoint>'
# Specify the prefix of the object that you want to upload to OSS.
upload_dir = 'user-dir-prefix/'
# Specify the validity period of the signature and the policy. Unit: seconds.
expire_time = 3600
def generate_expiration(seconds):
"""
The expiration time is calculated based on the specified validity period. Unit: seconds.
:param seconds: validity period (seconds).
:return: time string in the ISO 8601 standard. Example: 2014-12-01T12:00:00.000Z.
"""
now = int(time.time())
expiration_time = now + seconds
gmt = datetime.datetime.utcfromtimestamp(expiration_time).isoformat()
gmt += 'Z'
return gmt
def generate_signature(access_key_secret, expiration, conditions, policy_extra_props=None):
"""
Generate a signature string.
:param access_key_secret: the AccessKey secret that has the permissions to access the bucket.
:param expiration: the time when the signature expires. Specify the time in the ISO 8601 standard in the yyyy-MM-ddTHH:mm:ssZ format. Example: 2014-12-01T12:00:00.000Z.
:param conditions: the policy conditions that can be used to limit the values that you can set during form upload.
:param policy_extra_props: the additional policy parameters. If additional parameters are supported by the policy, you can pass additional parameters by using dict.
:return: the signature string.
"""
policy_dict = {
'expiration': expiration,
'conditions': conditions
}
if policy_extra_props is not None:
policy_dict.update(policy_extra_props)
policy = json.dumps(policy_dict).strip()
policy_encode = base64.b64encode(policy.encode())
h = hmac.new(access_key_secret.encode(), policy_encode, sha)
sign_result = base64.b64encode(h.digest()).strip()
return sign_result.decode()
def generate_upload_params():
policy = {
# The validity period of the policy.
"expiration": generate_expiration(expire_time),
# Specify the conditions in the policy field.
"conditions": [
# If success_action_redirect is not specified, the HTTP status code returned after a successful upload is 204.
["eq", "$success_action_status", "200"],
# The value of the form field must start with the specified prefix. For example, if the value of the key form field starts with user/user1, the condition is ["starts-with", "$key", "user/user1"].
["starts-with", "$key", upload_dir],
# Specify the minimum and maximum sizes of the object that you want to upload. Unit: bytes.
["content-length-range", 1, 1000000],
# Set the type of the object that you want to upload to the specified image type.
["in", "$content-type", ["image/jpg", "image/png"]]
]
}
signature = generate_signature(access_key_secret, policy.get('expiration'), policy.get('conditions'))
response = {
'policy': base64.b64encode(json.dumps(policy).encode('utf-8')).decode(),
'ossAccessKeyId': access_key_id,
'signature': signature,
'host': host,
'dir': upload_dir
# Specify additional parameters.
}
return json.dumps(response)
Sample code for the client
The following sample code provides an example on how to use the signature and policy to upload an object to OSS from a web client:
const form = document.querySelector('form');
const fileInput = document.querySelector('#file');
form.addEventListener('submit', (event) => {
event.preventDefault();
let file = fileInput.files[0];
let filename = fileInput.files[0].name;
fetch('/get_post_signature_for_oss_upload', { method: 'GET' })
.then(response => response.json())
.then(data => {
const formData = new FormData();
formData.append('name',filename);
formData.append('policy', data.policy);
formData.append('OSSAccessKeyId', data.ossAccessKeyId);
formData.append('success_action_status', '200');
formData.append('signature', data.signature);
formData.append('key', data.dir + filename);
// file must be the last form field. No order is required for other form fields.
formData.append('file', file);
fetch(data.host, { method: 'POST', body: formData },).then((res) => {
console.log(res);
alert ('Object Uploaded');
});
})
.catch(error => {
console.log('Error occurred while getting OSS upload parameters:', error);
});
});
Obtain a signed URL from the application server
The following figure shows how to upload an object to OSS from a client by using a signed URL obtained from the application server.
The client requests a signed URL from the application server.
The application server uses OSS SDKs to generate a signed URL for PUT requests and then returns the signed URL to the client.
The client uses the signed URL to call the PutObject operation to upload an object to OSS.
The following sample code provides an example on how to obtain a signed URL from the application server:
import oss2
from oss2.credentials import EnvironmentVariableCredentialsProvider
# Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured.
auth = oss2.ProviderAuth(EnvironmentVariableCredentialsProvider())
# Replace <your-endpoint> with the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com.
# Replace <your-bucket> with the name of the bucket.
bucket = oss2.Bucket(auth, '<your-endpoint>', '<your-bucket>')
# Specify the validity period. Unit: seconds.
expire_time = 3600
# Specify the full path of the object. Example: exampledir/exampleobject.png. Do not include the bucket name in the full path.
object_name = 'exampledir/exampleobject.png'
def generate_presigned_url():
# Specify headers.
headers = dict()
# Specify the Content-Type header.
headers['Content-Type'] = 'image/png'
# Specify the storage class of the object.
# headers["x-oss-storage-class"] = "Standard"
# By default, OSS identifies the forward slashes (/) in the full path of an object as escape characters when a signed URL is generated. Therefore, the signed URL cannot be used directly.
# Set the slash_safe parameter to True. This way, OSS does not identify the forward slashes (/) in the full path of the object as escape characters. Then, you can use the generated signed URL to upload the object.
url = bucket.sign_url('PUT', object_name, expire_time, slash_safe=True, headers=headers)
return url
Sample code for the client
The following sample code provides an example on how to use the signed URL to upload an object to OSS from a web client: