All Products
Search
Document Center

Object Storage Service:Upload objects directly to OSS from clients

Last Updated:Mar 08, 2024

Direct upload allows you to upload data directly to Object Storage Service (OSS) from clients. The direct upload solution accelerates uploads and reduces the resource usage of the application server by eliminating the need to transfer objects to and from the application server. This topic describes the benefits, implementation methods, and best practices of this solution.

Benefits

In the traditional server and client architecture, uploading data to OSS from the application server is the common upload method. The client uploads objects to the application server, and then the application server uploads the objects to OSS. During the upload process, data must be transferred twice over the network. This results in a waste of network resources and an increase in the resource usage of the application server. To resolve this issue, you can upload objects directly to OSS without the need to upload objects to the application server.

image

Implementation methods

To upload data directly to OSS from clients, you must resolve the following issues:

CORS

If your client is a web client or mini program, you must allow cross-origin resource sharing (CORS). To ensure data security, browsers and mini programs usually prohibit CORS. This also prevents your client code from being used to connect to OSS. You can configure CORS rules for an OSS bucket to allow the domain name of your web client or mini program to access OSS. For more information about how to configure CORS rules, see CORS.

Security authorization

To upload an object to OSS, you need to use the AccessKey pair of a RAM user to complete signature authentication. If the client uses a permanent AccessKey pair, the AccessKey pair may be leaked. This may cause security risks. To resolve this issue, you can use one of the following solutions to implement secure upload:

  • Obtain temporary access credentials from STS on the application server

    In most object upload scenarios, we recommend that you use Security Token Service (STS) SDKs to obtain temporary access credentials on the application server, and then use the temporary access credentials and OSS SDKs to upload objects from the client directly to OSS. The client can reuse the temporary access credentials on the application server to generate a signature. This is suitable for scenarios in which multipart upload and resumable upload are used to upload large objects. However, frequent calls to STS may cause throttling. In this case, we recommend that you cache the temporary access credentials and renew them before the validity period ends. For more information, see What is STS?

  • Obtain a signature and POST policy for PostObject from the application server

    In scenarios in which you want to limit the attributes of an object to upload, you need to obtain information required to call the PostObject operation from the application server. The required information includes a signature and a POST policy. The client can then use the obtained information to upload the object directly to OSS without using OSS SDKs. You can use the policy generated by the server to limit the attributes of an object that can be uploaded, such as the object size and type. This solution is suitable for uploading objects by using HTML forms. This solution does not support multipart upload and resumable upload. For more information, see PostObject.

  • Obtain a signed URL for PutObject from the application server

    For simple object upload scenarios, you can use OSS SDKs on the application server to obtain a signed URL that is required to call the PutObject operation. Then, the client can use the signed URL to upload an object without using OSS SDKs. This solution is not suitable for multipart upload and resumable upload. The application server generates a signed URL for each part and returns the signed URL to the client. This increases the number of interactions with the application server and the complexity of network requests. In addition, the client may modify the content or order of the parts, which results in an invalid combined object. For more information, see Create a signed URL by using signature V1.

Obtain temporary access credentials from STS on the application server

The following figure shows how to upload an object to OSS from a client by using STS temporary access credentials returned from the application server.

image
  1. The client requests temporary access credentials from the application server.

  2. The application server uses STS SDKs to call the AssumeRole operation to obtain temporary access credentials.

  3. STS generates and returns the temporary access credentials to the application server.

  4. The application server returns the temporary access credentials to the client.

  5. The client uses OSS SDKs to upload an object to OSS by using the temporary access credentials.

  6. OSS returns a success response to the client.

Demo project

sts.zip

Examples

Server-side sample code

The following sample code provides an example on how the application server obtains temporary access credentials from STS:

import json
from alibabacloud_tea_openapi.models import Config
from alibabacloud_sts20150401.client import Client as Sts20150401Client
from alibabacloud_sts20150401 import models as sts_20150401_models
from alibabacloud_credentials.client import Client as CredentialClient

# Replace <YOUR_ROLE_ARN> with the Alibaba Cloud Resource Name (ARN) of the RAM role that has the permissions to upload objects to the OSS bucket. 
role_arn_for_oss_upload = '<YOUR_ROLE_ARN>'
# Replace <YOUR_REGION_ID> with the ID of the region of the STS service. Example: cn-hangzhou. 
region_id = '<YOUR_REGION_ID>'

def get_sts_token():
    # If you do not specify parameters when you initialize CredentialClient, the default credential chain is used. 
    # If you run the program on your local computer, you can specify the AccessKey pair by configuring the ALIBABA_CLOUD_ACCESS_KEY_ID and ALIBABA_CLOUD_ACCESS_KEY_SECRET environment variables.
    # If you run the program on an Elastic Compute Service (ECS) instance, an elastic container instance, or a node in a Container Service for Kubernetes (ACK) cluster, you can specify the bound instance or node by configuring the ALIBABA_CLOUD_ECS_METADATA environment variable. The SDK automatically exchanges the temporary access credentials obtained from STS. 
    config = Config(region_id=region_id, credential=CredentialClient())
    sts_client = Sts20150401Client(config=config)
    assume_role_request = sts_20150401_models.AssumeRoleRequest(
        role_arn=role_arn_for_oss_upload,
        # Replace <YOUR_ROLE_SESSION_NAME> with a custom session name, such as oss-role-session. 
        role_session_name='<YOUR_ROLE_SESSION_NAME>'
    )
    response = sts_client.assume_role(assume_role_request)
    token = json.dumps(response.body.credentials.to_map())
    return token

Client-side sample code

The following sample code provides an example on how to use the temporary access credentials to upload an object to OSS from a web client:

let credentials = null;
const form = document.querySelector("form");
form.addEventListener("submit", async (event) => {
  event.preventDefault();
  // Obtain temporary access credentials again only if they expire. This reduces the number of calls to STS. 
  if (isCredentialsExpired(credentials)) {
    const response = await fetch("/get_sts_token_for_oss_upload", {
      method: "GET",
    });
    credentials = await response.json();
  }
  const client = new OSS({
    // Replace <YOUR_BUCKET> with the name of the OSS bucket. 
    bucket: "<YOUR_BUCKET>",
    // Replace <YOUR_REGION> with the ID of the region in which the OSS bucket is located. Example: oss-cn-hangzhou. 
    region: "oss-<YOUR_REGION>",
    accessKeyId: credentials.AccessKeyId,
    accessKeySecret: credentials.AccessKeySecret,
    stsToken: credentials.SecurityToken,
  });

  const fileInput = document.querySelector("#file");
  const file = fileInput.files[0];
  const result = await client.put(file.name, file);
  console.log(result);
});

/**
 * Determine whether the temporary access credentials expire. 
 **/
function isCredentialsExpired(credentials) {
  if (!credentials) {
    return true;
  }
  const expireDate = new Date(credentials.Expiration);
  const now = new Date();
  // If the remaining validity period of the temporary access credentials is less than 1 minute, the temporary access credentials are considered expired. 
  return expireDate.getTime() - now.getTime() <= 60000;
}

Obtain a signature and POST policy for PostObject from the application server

The following figure shows how to upload an object from a client to OSS by using the signature and POST policy that are obtained from the application server.

image
  1. The client requests information, such as a signature and POST policy, from the application server.

  2. The application server generates and returns information, such as the signature and POST policy, to the client.

  3. The client uses information, such as the signature and POST policy, to call the PostObject operation to upload the object to OSS by using an HTML form.

  4. OSS returns a success response to the client.

Demo project

postsignature.zip

Examples

Server-side sample code

The following sample code provides an example on how to obtain a signature and POST policy from the application server:

import os
from hashlib import sha1 as sha
import json
import base64
import hmac
import datetime
import time

# Configure the OSS_ACCESS_KEY_ID environment variable. 
access_key_id = os.environ.get('OSS_ACCESS_KEY_ID')
# Configure the OSS_ACCESS_KEY_SECRET environment variable. 
access_key_secret = os.environ.get('OSS_ACCESS_KEY_SECRET')
# Replace <YOUR_BUCKET> with the name of the bucket. 
bucket = '<YOUR_BUCKET>'
# Set host to a value that is in the BucketName.Endpoint format. Replace <YOUR_BUCKET> with the name of the bucket. Replace <YOUR_ENDPOINT> with the endpoint of the region in which the bucket is located. 
host = 'https://<YOUR_BUCKET>.<YOUR_ENDPOINT>'
# Specify the prefix in the name of the object that you want to upload to OSS. 
upload_dir = 'user-dir-prefix/'
# Specify the validity period. Unit: seconds. 
expire_time = 3600


def generate_expiration(seconds):
    """
    The expiration time is calculated based on the specified validity period. Unit: seconds. 
    :param seconds: the validity period (seconds). 
    :return: the time string in the ISO 8601 standard. Example: 2014-12-01T12:00:00.000Z. 
    """
    now = int(time.time())
    expiration_time = now + seconds
    gmt = datetime.datetime.utcfromtimestamp(expiration_time).isoformat()
    gmt += 'Z'
    return gmt


def generate_signature(access_key_secret, expiration, conditions, policy_extra_props=None):
    """
    Generate a signature string. 
    :param access_key_secret: the AccessKey secret that has the permissions to access the bucket. 
    :param expiration: the time when the signature expires. Specify the time in the ISO 8601 standard in the yyyy-MM-ddTHH:mm:ssZ format. Example: 2014-12-01T12:00:00.000Z. 
    :param conditions: the policy conditions that can be used to limit the values that you can specify during form upload. 
    :param policy_extra_props: the additional policy parameters. You can pass additional parameters as a dict. 
    :return: the signature string. 
    """
    policy_dict = {
        'expiration': expiration,
        'conditions': conditions
    }
    if policy_extra_props is not None:
        policy_dict.update(policy_extra_props)
    policy = json.dumps(policy_dict).strip()
    policy_encode = base64.b64encode(policy.encode())
    h = hmac.new(access_key_secret.encode(), policy_encode, sha)
    sign_result = base64.b64encode(h.digest()).strip()
    return sign_result.decode()

def generate_upload_params():
    policy = {
        # Specify the validity period of the policy. 
        "expiration": generate_expiration(expire_time),
        # Specify the policy conditions. 
        "conditions": [
            # If success_action_redirect is not specified, HTTP status code 204 is returned after the object is uploaded. 
            ["eq", "$success_action_status", "200"],
            # The value of the form field must start with the specified prefix. For example, if the value of the key form field starts with user/user1, the condition is ["starts-with", "$key", "user/user1"]. 
            ["starts-with", "$key", upload_dir],
            # Specify the minimum and maximum sizes of the object that can be uploaded. Unit: bytes. 
            ["content-length-range", 1, 1000000],
            # Set the type of the object that you want to upload to the specified image type.
            ["in", "$content-type", ["image/jpg", "image/png"]]
        ]
    }
    signature = generate_signature(access_key_secret, policy.get('expiration'), policy.get('conditions'))
    response = {
        'policy': base64.b64encode(json.dumps(policy).encode('utf-8')).decode(),
        'ossAccessKeyId': access_key_id,
        'signature': signature,
        'host': host,
        'dir': upload_dir
        # Specify additional parameters.
    }
    return json.dumps(response)

Client-side sample code

The following sample code provides an example on how to use the signature and POST policy to upload an object to OSS from a web client:

const form = document.querySelector('form');
const fileInput = document.querySelector('#file');
form.addEventListener('submit', (event) => {
  event.preventDefault();
  let file = fileInput.files[0];
  let filename = fileInput.files[0].name;
  fetch('/get_post_signature_for_oss_upload', { method: 'GET' })
    .then(response => response.json())
    .then(data => {
      const formData = new FormData();
      formData.append('name',filename);
      formData.append('policy', data.policy);
      formData.append('OSSAccessKeyId', data.ossAccessKeyId);
      formData.append('success_action_status', '200');
      formData.append('signature', data.signature);
      formData.append('key', data.dir + filename);
      // file must be the last form field. No particular order is required for other form fields. 
      formData.append('file', file);
      fetch(data.host, { method: 'POST', body: formData },).then((res) => {
        console.log(res);
        alert ('Object Uploaded');
      });
    })
    .catch(error => {
      console.log('Error occurred while getting OSS upload parameters:', error);
    });
});

Obtain a signed URL for PutObject from the application server

The following figure shows how to upload an object to OSS from a client by using a signed URL obtained from the application server.

image
  1. The client requests a signed URL from the application server.

  2. The application server uses OSS SDKs to generate a signed URL for PUT requests and then returns the signed URL to the client.

  3. The client uses the signed URL to call the PutObject operation to upload an object to OSS.

  4. OSS returns a success response to the client.

Demo project

presignedurl.zip

Examples

Server-side sample code

The following sample code provides an example on how to obtain a signed URL from the application server:

import oss2
from oss2.credentials import EnvironmentVariableCredentialsProvider

# Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
auth = oss2.ProviderAuth(EnvironmentVariableCredentialsProvider())
# Replace <YOUR_ENDPOINT> with the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. 
# Replace <YOUR_BUCKET> with the name of the bucket. 
bucket = oss2.Bucket(auth, '<YOUR_ENDPOINT>', '<YOUR_BUCKET>')
# Specify the validity period. Unit: seconds. 
expire_time = 3600
# Specify the full path of the object. Example: exampledir/exampleobject.png. Do not include the bucket name in the full path. 
object_name = 'exampledir/exampleobject.png'

def generate_presigned_url():
    # Specify the headers. 
    headers = dict()
    # Specify the Content-Type header. 
    headers['Content-Type'] = 'image/png'
    # Specify the storage class. 
    # headers["x-oss-storage-class"] = "Standard"
    # By default, OSS identifies the forward slashes (/) in the full path of an object as escape characters when a signed URL is generated. In this case, you cannot use the signed URL directly. 
    # Set the slash_safe parameter to True. This way, OSS does not identify the forward slashes (/) in the full path of the object as escape characters. In this case, you can use the generated signed URL to upload the object. 
    url = bucket.sign_url('PUT', object_name, expire_time, slash_safe=True, headers=headers)
    return url

Client-side sample code

The following sample code provides an example on how to use the signed URL to upload an object to OSS from a web client:

const form = document.querySelector("form");
form.addEventListener("submit", (event) => {
  event.preventDefault();
  const fileInput = document.querySelector("#file");
  const file = fileInput.files[0];
  fetch(`/get_presigned_url_for_oss_upload?filename=${file.name}`, { method: "GET" })
    .then((response) => {
        return response.text();
     })
    .then((url) => {
      fetch(url, {
        method: "PUT",
        headers: new Headers({
          'Content-Type': 'image/png',
        }),
        body: file,
       }).then((res) => {
            console.log(res);
            alert ('Object Uploaded');
       });
   });
});

Best practices

For information about best practices for uploading data to OSS from clients, see the following topics: