Mirroring-based back-to-origin - Object Storage Service - Alibaba Cloud Documentation Center

After you configure mirroring-based back-to-origin rules for a bucket, if a requested object does not exist in the bucket, Object Storage Service (OSS) obtains the object from the origin specified by the back-to-origin rules. OSS returns the object obtained from the origin to the requester and stores the object in the bucket.

Limits

Supported regions
You can configure mirroring-based back-to-origin rules for buckets that are located in the following regions: China (Hangzhou), China (Shanghai), China (Qingdao), China (Beijing), China (Zhangjiakou), China (Hohhot), China (Ulanqab), China (Shenzhen), China (Heyuan), China (Guangzhou), China (Chengdu), China (Hong Kong), US (Silicon Valley), US (Virginia), Japan (Tokyo), Singapore, Australia (Sydney), Malaysia (Kuala Lumpur), Indonesia (Jakarta), Philippines (Manila), India (Mumbai), Germany (Frankfurt), UK (London), and UAE (Dubai).
Number of back-to-origin rules
You can configure up to 20 back-to-origin rules for a bucket. The rules are sequentially matched with requests based on the values of RuleNumber. If a request matches a rule, the subsequent rules are not matched. OSS determines whether a request matches a back-to-origin rule based on whether the request meets the conditions specified in the rule. OSS does not check whether the requested object can be obtained from the origin.
QPS and traffic limits
In regions in the Chinese mainland, the default queries per second (QPS) for mirroring-based back-to-origin requests is 2,000 and the default bandwidth is 2 Gbit/s. In regions outside the Chinese mainland, the default QPS for mirroring-based back-to-origin requests is 1,000 and the default bandwidth is 1 Gbit/s.
If your business requires higher QPS or bandwidth, contact technical support.
Back-to-origin URL
An internal endpoint cannot be used as a back-to-origin URL.
Default timeout period
By default, the timeout period of a mirroring-based back-to-origin request is 10 seconds.

Scenarios

Mirroring-based back-to-origin allows you to seamlessly migrate data to OSS. For example, you can use mirroring-based back-to-origin to migrate services from a self-managed origin or from another cloud service to OSS without service interruption. For more information about use cases, see Seamlessly migrate data of a web-based service provider to OSS.

Procedure

The following figure shows how mirroring-based back-to-origin works.

Configure back-to-origin rules

Trigger conditions
OSS obtains an object from the origin based on the mirroring-based back-to-origin rule only if HTTP status code 404 is returned for the GetObject operation.
Naming conventions
The URL that OSS uses to obtain an object from the origin is in the following format: http(s)://MirrorURL/ObjectName. ObjectName specifies the name of the requested object. For example, if the back-to-origin URL configured for a bucket is https://aliyun.com and the requested object named example.jpg does not exist in the bucket, OSS obtains the object from https://aliyun.com/example.jpg and stores the object as example.jpg.
Rules for failed back-to-origin requests
If the requested object is not found at the origin, the origin returns HTTP status code 404 to OSS. Then, OSS returns the same HTTP status code to the requester. If the origin returns a non-200 HTTP status code, which indicates an error, such as an object retrieval failure due to network errors, OSS returns 424 MirrorFailed to the requester.
Rules to update objects obtained from the origin
After OSS obtains an object by using mirroring-based back-to-origin, if the object at the origin is modified, OSS does not update the obtained object.
Metadata of objects obtained from the origin
OSS stores the following HTTP headers that are returned from the origin as the object metadata:
```
Content-Type
Content-Encoding
Content-Disposition
Cache-Control
Expires
Content-Language
Access-Control-Allow-Origin
```
HTTP request rules
- Headers that are contained in the request sent to OSS are not contained in the request sent by OSS to the origin. OSS determines whether to send the QueryString information to the origin based on the back-to-origin rules that you configure.
- If the origin returns chunked-encoded data, OSS returns chunked-encoded data.

Use the OSS console

If you configure multiple back-to-origin rules for a bucket in the OSS console, the rules are used to match a request based on the sequence in which the rules are configured. You can click Move Up or Move Down on the right side of the rules to change the priority of the rules.

When a requester requests an object that does not exist in the bucket, you can specify the URL of the object in the origin and back-to-origin conditions to obtain the object. For example, you have a bucket named examplebucket in the China (Hangzhou) region and want OSS to search the examplefolder directory at the https://www.example.com/ origin for the requested object and return the object to the requester when the requested object does not exist in the examplefolder directory of the root directory in the bucket. In this case, you can perform the following steps to configure a back-to-origin rule:

Log on to the OSS console.
In the left-side navigation pane, click Buckets. On the Buckets page, click the name of the desired bucket.
In the navigation tree, choose Data Management > Back-to-Origin.
On the Back-to-Origin page, click Create Rule.

In the Create Rule panel, configure the parameters. The following table describes the parameters. Retain the default configurations of other parameters.


Parameter	Description
Mode	Select Mirroring.
Prerequisite	Select File Name Prefix and set File Name Prefix to examplefolder/. Note When you configure only one back-to-origin rule, you can leave the File Name Prefix parameter and the File Name Suffix parameter empty. When you configure multiple back-to-origin rules, you must set different file name prefixes or suffixes to differentiate back-to-origin rules.
Origin URL	Select https in the first text box, enter www.example.com in the second text box, and enter examplefolder in the third text box.

Click OK.
The following section describes the access process after the preceding back-to-origin rule is configured:
1. A requester requests https://examplebucket.oss-cn-hangzhou.aliyuncs.com/examplefolder/example.txt for the first time.
2. If the examplefolder/example.txt object does not exist in examplebucket, OSS sends a request to the origin by using https://www.example.com/examplefolder/example.txt.
3. If the required object is obtained, OSS writes the example.txt object to the examplefolder directory in examplebucket and then returns the object to the requester. If the required object is not obtained, HTTP status code 404 is returned to the requester.

You can follow the preceding steps to configure mirroring-based back-to-origin rules for basic usage scenarios. For more information about how to configure mirroring-based back-to-origin rules for specific scenarios, see Configuration examples of mirroring-based back-to-origin.

Use OSS SDKs

The following sample code provides examples on how to configure mirroring-based back-to-origin rules by using OSS SDKs for common programming languages. For more information about the sample code for configuring mirroring-based back-to-origin rules by using OSS SDKs for other programming languages, see Overview.

Java

import com.aliyun.oss.ClientException;
import com.aliyun.oss.OSS;
import com.aliyun.oss.OSSClientBuilder;
import com.aliyun.oss.OSSException;
import com.aliyun.oss.model.RoutingRule;
import com.aliyun.oss.model.SetBucketWebsiteRequest;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

public class Demo {

    public static void main(String[] args) throws Exception {
        // In this example, the endpoint of the China (Hangzhou) region is used. Specify your actual endpoint. 
        String endpoint = "https://oss-cn-hangzhou.aliyuncs.com";
        // The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. Using these credentials to perform operations in OSS is a high-risk operation. We recommend that you use a RAM user to call API operations or perform routine O&M. To create a RAM user, log on to the RAM console. 
        String accessKeyId = "yourAccessKeyId";
        String accessKeySecret = "yourAccessKeySecret";
        // Specify the name of the bucket. Example: examplebucket. 
        String bucketName = "examplebucket";

        // Create an OSSClient instance. 
        OSS ossClient = new OSSClientBuilder().build(endpoint, accessKeyId, accessKeySecret);

        try {
            SetBucketWebsiteRequest request = new SetBucketWebsiteRequest(bucketName);
            // Specify the operation that you want to perform if the default homepage is specified, the name of the requested object does not end with a forward slash (/), and the object does not exist. 
            //request.setSubDirType(null);
            // Specify whether to redirect the requester to the default homepage in the subdirectory when the subdirectory is accessed. 
            //request.setSupportSubDir(false);

            List<RoutingRule> routingRules = new ArrayList<RoutingRule>();

            RoutingRule rule = new RoutingRule();
            rule.setNumber(1);
            // Specify the prefix of object names. Only the objects whose names contain the specified prefix match the rule. 
            rule.getCondition().setKeyPrefixEquals("examplebucket");
            // Specify the HTTP status code. The rule is matched only when the specified object is accessed and HTTP status code 404 is returned. 
            rule.getCondition().setHttpErrorCodeReturnedEquals(404);
            // Specify the redirection type. 
            rule.getRedirect().setRedirectType(RoutingRule.RedirectType.Mirror);
            // Specify the URL of the origin for mirroring-based back-to-origin. Example: https://www.example.com/. 
            rule.getRedirect().setMirrorURL("<yourMirrorURL>");
            //rule.getRedirect().setMirrorRole("AliyunOSSMirrorDefaultRole");
            // Specify whether to include the request parameters when the redirection rule or the mirroring-based back-to-origin rule is executed. 
            rule.getRedirect().setPassQueryString(true);
            // This parameter is used in the same manner as the PassQueryString parameter and is assigned a higher priority level than the PassQueryString parameter. This parameter takes effect only when RedirectType is set to Mirror. 
            rule.getRedirect().setMirrorPassQueryString(true);
            // Specify the HTTP status code in the response. This parameter takes effect only when RedirectType is set to External or AliCDN. 
            //rule.getRedirect().setHttpRedirectCode(302);
            // Specify the domain name that you want to use for redirection. The domain name must comply with the naming conventions for domains. 
            //rule.getRedirect().setHostName("oss.aliyuncs.com");
            // Specify the protocol that is used for redirection. This parameter takes effect only when RedirectType is set to External or AliCDN. 
            //rule.getRedirect().setProtocol(RoutingRule.Protocol.Https);
            // Specify the string that you want to use to replace the object name when the request is redirected. You can set this parameter to a variable. 
            //rule.getRedirect().setReplaceKeyWith("${key}.jpg");
            // If this parameter is set to true, the prefix of the object name is replaced with the value specified by ReplaceKeyPrefixWith. 
            rule.getRedirect().setEnableReplacePrefix(true);
            // Specify the string that you want to use to replace the prefix of the object name during redirection. 
            rule.getRedirect().setReplaceKeyPrefixWith("examplebucket");
            // Specify whether to check the MD5 hash of the response body that is returned by the origin. This parameter takes effect only when RedirectType is set to Mirror. 
            rule.getRedirect().setMirrorCheckMd5(true);

            RoutingRule.MirrorHeaders mirrorHeaders = new RoutingRule.MirrorHeaders();
            // Specify whether to pass through all request headers to the origin. This parameter takes effect only when RedirectType is set to Mirror. 
            mirrorHeaders.setPassAll(false);
            List passes = new ArrayList<String>();
            passes.add("cache-control");
            // Specify the headers that you want to pass through to the origin. This parameter takes effect only when RedirectType is set to Mirror. 
            mirrorHeaders.setPass(passes);
            List removes = new ArrayList<String>();
            removes.add("content-type");
            // Specify the headers that you do not want to pass through to the origin. This parameter takes effect only when RedirectType is set to Mirror. 
            mirrorHeaders.setRemove(removes);
            List sets = new ArrayList<Map<String, String>>();
            Map header1 = new HashMap<String, String>();
            header1.put("Key", "key1");
            header1.put("Value", "value1");
            Map header2 = new HashMap<String, String>();
            header2.put("Key", "key2");
            header2.put("Value", "value2");
            sets.add(header1);
            sets.add(header2);
            // Specify the headers that you want to pass through to the origin. The specified headers are passed to the origin regardless of whether they are included in the request. 
            mirrorHeaders.setSet(sets);
            // Specify the headers that you want to include in the response when the requested object is obtained from the origin. This parameter takes effect only when RedirectType is set to Mirror. 
            rule.getRedirect().setMirrorHeaders(mirrorHeaders);

            routingRules.add(rule);
            request.setRoutingRules(routingRules);
            ossClient.setBucketWebsite(request);
        } catch (OSSException oe) {
            System.out.println("Caught an OSSException, which means your request made it to OSS, "
                    + "but was rejected with an error response for some reason.");
            System.out.println("Error Message:" + oe.getErrorMessage());
            System.out.println("Error Code:" + oe.getErrorCode());
            System.out.println("Request ID:" + oe.getRequestId());
            System.out.println("Host ID:" + oe.getHostId());
        } catch (ClientException ce) {
            System.out.println("Caught an ClientException, which means the client encountered "
                    + "a serious internal problem while trying to communicate with OSS, "
                    + "such as not being able to access the network.");
            System.out.println("Error Message:" + ce.getMessage());
        } finally {
            if (ossClient != null) {
                ossClient.shutdown();
            }
        }
    }
}

Python

# -*- coding: utf-8 -*-
import oss2
from oss2.models import *

# The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. Using these credentials to perform operations in OSS is a high-risk operation. We recommend that you use a RAM user to call API operations or perform routine O&M. To create a RAM user, log on to the RAM console. 
auth = oss2.Auth('yourAccessKeyId', 'yourAccessKeySecret')
# In this example, the endpoint of the China (Hangzhou) region is used. Specify your actual endpoint. 
# Specify the name of the bucket. Example: examplebucket. 
bucket = oss2.Bucket(auth, 'https://oss-cn-hangzhou.aliyuncs.com', 'examplebucket')

# Enable static website hosting, set the default homepage to index.html, and then set the default 404 page to error.html. 
index_file = 'index.html'
error_file = 'error.html'
# Specify the match conditions. 
condition1 = Condition(key_prefix_equals='examplefolder',
                       http_err_code_return_equals=404)

# Specify the headers that you want to include in the request when you use mirroring-based back-to-origin. 
mirror_headers_set_1 = MirrorHeadersSet("myheader-key5","myheader-value5")
mirror_headers_set_2 = MirrorHeadersSet("myheader-key6","myheader-value6")
set_list = [mirror_headers_set_1, mirror_headers_set_2]
pass_list = ['myheader-key1', 'myheader-key2']
remove_list = ['myheader-key3', 'myheader-key4']
mirror_header = RedirectMirrorHeaders(pass_all=True, pass_list=pass_list, remove_list=remove_list, set_list=set_list)
# Specify the operation to perform after the rule is matched. 
redirect1 = Redirect(redirect_type=REDIRECT_TYPE_MIRROR, mirror_url='https://www.example.com/',
                     mirror_pass_query_string=True, mirror_follow_redirect=True, mirror_check_md5=True, mirror_headers=mirror_header)

rule1 = RoutingRule(rule_num=1, condition=condition1, redirect=redirect1)
website_set = BucketWebsite(index_file, error_file, [rule1])

# Configure mirroring-based back-to-origin. 
bucket.put_bucket_website(website_set)

package main

import (
    "fmt"
    "os"
    "github.com/aliyun/aliyun-oss-go-sdk/oss"
)

func main() {
    // Create an OSSClient instance. 
    // Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. Specify your actual endpoint. 
    // The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. Using these credentials to perform operations in OSS is a high-risk operation. We recommend that you use a RAM user to call API operations or perform routine O&M. To create a RAM user, log on to the RAM console. 
    client, err := oss.New("yourEndpoint", "yourAccessKeyId", "yourAccessKeySecret")
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }

    // Specify the name of the bucket. Example: examplebucket. 
    bucketName := "examplebucket"
    putXml := `<WebsiteConfiguration>
      <IndexDocument>
        <Suffix>index.html</Suffix>
        <SupportSubDir>true</SupportSubDir>
        <Type>0</Type>
      </IndexDocument>
      <ErrorDocument>
        <Key>error.html</Key>
        <HttpStatus>404</HttpStatus>
      </ErrorDocument>
      <RoutingRules>
        <RoutingRule>
          <RuleNumber>1</RuleNumber>
          <Condition>
            <! -- Specify the prefix of object names. Only the objects whose names contain the specified prefix match the rule. -->
            <KeyPrefixEquals>examplefolder/</KeyPrefixEquals>
            <! -- Specify the HTTP status code. The rule is matched only if the specified object is requested and the HTTP status code 404 is returned. -->
            <HttpErrorCodeReturnedEquals>404</HttpErrorCodeReturnedEquals>
          </Condition>
          <Redirect>
            <! -- Specify the redirection type. -->
            <RedirectType>Mirror</RedirectType>
            <PassQueryString>true</PassQueryString>
            <! -- Specify the back-to-origin URL for a mirroring-based back-to-origin rule. In this example, the back-to-origin URL is https://www.example.com/. -->
            <MirrorURL>http://example.com/</MirrorURL>-->           
            <! -- Specify whether to include the request parameters when the redirection rule or the mirroring-based back-to-origin rule is triggered. -->
            <PassQueryString>true</PassQueryString>
            <! -- Specify whether to use MirrorPassQueryString. This parameter is used in the same manner as the PassQueryString parameter and is assigned a higher priority than the PassQueryString parameter. This parameter takes effect only when RedirectType is set to Mirror. -->
            <MirrorPassQueryString>true</MirrorPassQueryString>
            <! -- Specify the HTTP status code in the response. This parameter takes effect only when RedirectType is set to External or AliCDN. -->
            <HttpRedirectCode>302</HttpRedirectCode>
            <! -- Specify the domain name that you want to use for redirection. The domain name must comply with the naming conventions for domains. -->
            <HostName>oss.aliyuncs.com</HostName>
            <! -- Specify the protocol that you want to use for redirection. This parameter takes effect only when RedirectType is set to External or AliCDN. -->
            <Protocol>https</Protocol>
            <! -- Specify the string that you want to use to replace the object name when the request is redirected. You can set this parameter to a variable. -->
            <ReplaceKeyWith>key.jpg</ReplaceKeyWith>
            <! -- If this parameter is set to true, the prefix of the object name is replaced with the value specified by ReplaceKeyPrefixWith. -->
            <EnableReplacePrefix>true</EnableReplacePrefix>
            <! -- Specify the string that you want to use to replace the prefix of the object name when the request is redirected. -->
            <ReplaceKeyPrefixWith>examplebucket</ReplaceKeyPrefixWith>
            <! -- Specify whether to check the MD5 hash of the body of the response that is returned by the origin. This parameter takes effect only when RedirectType is set to Mirror. -->
            <MirrorCheckMd5>false</MirrorCheckMd5>
            <! -- Specify whether to redirect the access to the address that is specified by Location if the origin returns HTTP status code 3xx.  This parameter takes effect only when RedirectType is set to Mirror. -->
            <MirrorFollowRedirect>true</MirrorFollowRedirect>
            <MirrorHeaders>
              <! -- Specify whether to pass through all request headers except the following header to the origin. This parameter takes effect only when RedirectType is set to Mirror. -->
              <PassAll>true</PassAll>
              <! -- Specify the headers that you want to pass through to the origin. This parameter takes effect only when RedirectType is set to Mirror. -->
              <Pass>myheader-key1</Pass>
              <Pass>myheader-key2</Pass>
              <! -- Specify the headers that you do not want to pass through to the origin. This parameter takes effect only when RedirectType is set to Mirror. -->
              <Remove>myheader-key3</Remove>
              <Remove>myheader-key4</Remove>
               <! -- Specify the headers that you want to pass through to the origin. The specified headers are passed to the origin regardless of whether the headers are included in the request. -->
              <Set>
                <Key>myheader-key5</Key>
                <Value>myheader-value5</Value>
              </Set>
            </MirrorHeaders>
          </Redirect>
        </RoutingRule>
      </RoutingRules>
    </WebsiteConfiguration>    `

    err = client.SetBucketWebsiteXml(bucketName,putXml)
    if err != nil {
        fmt.Println("Error:", err)
        os.Exit(-1)
    }
}

Use ossutil

For more information about how to configure mirroring-based back-to-origin rules by using ossutil, see Add or modify website-related configurations.

Use RESTful APIs

If your business requires a high level of customization, you can directly call RESTful APIs. To directly call an API, you must include the signature calculation in your code. For more information, see PutBucketWebsite.