All Products
Search
Document Center

Object Storage Service:Mirroring-based back-to-origin

Last Updated:Feb 28, 2024

After you migrate all existing data from your server to an Object Storage Service (OSS) bucket, you can configure a mirroring-based back-to-origin rule to allow requesters to access data that is available on your server but has not been migrated to the bucket. When a requester requests a resource that does not exist in the bucket, OSS attempts to retrieve the resource from the origin server that is specified in the mirroring-based back-to-origin rule. If the requested resource is retrieved from the origin server, OSS stores the resource in the bucket and returns the resource to the requester.

Limits

  • Regions

    You can configure mirroring-based back-to-origin rules for buckets that are located in the following regions: China (Hangzhou), China (Shanghai), China (Qingdao), China (Beijing), China (Zhangjiakou), China (Hohhot), China (Ulanqab), China (Shenzhen), China (Heyuan), China (Guangzhou), China (Chengdu), China (Hong Kong), US (Silicon Valley), US (Virginia), Japan (Tokyo), Singapore, Australia (Sydney), Malaysia (Kuala Lumpur), Indonesia (Jakarta), Philippines (Manila), India (Mumbai), Germany (Frankfurt), UK (London), and UAE (Dubai).

  • Number of back-to-origin rules

    You can configure up to 20 back-to-origin rules for a bucket. The rules are sequentially matched with requests based on the values of RuleNumber. If a back-to-origin rule is matched, the subsequent rules are ignored. OSS determines whether a request matches a back-to-origin rule based on whether the request meets the conditions specified in the rule. OSS does not check whether the requested object can be obtained from the origin.

  • QPS and traffic limits

    In regions in the Chinese mainland, the default maximum queries per second (QPS) for mirroring-based back-to-origin requests and default maximum bandwidth for an Alibaba Cloud account are 2,000 and 2 Gbit/s. In regions outside the Chinese mainland, the default maximum QPS for mirroring-based back-to-origin requests and default maximum bandwidth for an Alibaba Cloud account are 1,000 and 1 Gbit/s.

    If your business requires higher QPS or bandwidth, contact technical support.

  • Origin URLs

    Origin URLs cannot be URLs on an internal network.

  • Default timeout period

    By default, the timeout period of a mirroring-based back-to-origin request is 10 seconds.

Scenarios

Mirroring-based back-to-origin is used to seamlessly migrate data to OSS. For example, you can use mirroring-based back-to-origin to migrate services from a self-managed origin or another cloud service to OSS without business interruption. During the migration, you can use mirroring-based back-to-origin to obtain the data that has not been migrated to OSS. This ensures business continuity. For more information about use cases, see Seamlessly migrate data of a web-based service provider to OSS.

Note

Mirroring-based back-to-origin does not incur additional fees. However, a mirroring-based back-to-origin request itself incurs a request fee for an API operation call. A mirroring-based back-to-origin request is billed the same way as other requests. For more information, see API operation calling fees.

Process

The following figure shows how mirroring-based back-to-origin works.

image

Back-to-origin rules

  • Trigger conditions

    OSS obtains an object from the origin based on the mirroring-based back-to-origin rule only if 404 is returned for the GetObject operation.

  • Rule naming

    The URL that OSS uses to obtain an object from the origin is in the following format: http(s)://MirrorURL/ObjectName. ObjectName specifies the name of the requested object. For example, if the back-to-origin URL configured for a bucket is https://aliyun.com and the requested object named example.jpg does not exist in the bucket, OSS obtains the object from https://aliyun.com/example.jpg and stores the object as example.jpg in OSS.

  • Failed back-to-origin requests

    If the requested object is not found at the origin, the origin returns HTTP 404 status code to OSS. Then, OSS returns the same HTTP status code to the requester. If the origin returns an HTTP status code other than 200 and 404, the back-to-origin request fails due to factors such as network errors. In this case, OSS returns 424 MirrorFailed to the requester.

  • Updates of objects obtained from an origin

    After OSS obtains an object by using mirroring-based back-to-origin, if the object at the origin is modified, OSS does not update the obtained object.

  • Metadata of objects obtained from an origin

    OSS stores the following HTTP headers returned by an origin server as the metadata of OSS objects:

    Content-Type
    Content-Encoding
    Content-Disposition
    Cache-Control
    Expires
    Content-Language
    Access-Control-Allow-Origin
  • HTTP requests

    • Headers that are contained in the request sent to OSS are not contained in the request sent by OSS to an origin. OSS determines whether to send the QueryString information to the origin based on the configured back-to-origin rules.

    • If the origin returns chunked-encoded data, OSS returns chunked-encoded data.

Procedure

Use the OSS console

If you configure multiple back-to-origin rules for a bucket in the OSS console, the rules apply in chronological order of rule creation by default. You can click Up or Down on the right side of the rules to change the priority of the rules.

3

When a requester requests an object that does not exist in the bucket, OSS attempts to retrieve the object from the origin server based on the configured origin URL and back-to-origin conditions. For example, you have the examplefolder subdirectory in the root directory of the examplebucket bucket in the China (Hangzhou) region. If a requested object does not exist in the examplefolder subdirectory, you want OSS to search the examplefolder directory at https://www.example.com/ for the requested object and return the object to the requester. To achieve this goal, perform the following steps to configure a back-to-origin rule:

  1. Log on to the OSS console.

  2. In the left-side navigation pane, click Buckets. On the Buckets page, find and click the desired bucket.

  3. In the left-side navigation tree, choose Data Management > Mirroring-based Back-to-origin.

  4. On the Mirroring-based Back-to-origin page, click Create Rule.

  5. In the Create Rule panel, configure the required parameters described in the following table and retain the default settings for other parameters.

    Parameter

    Description

    Method

    Select Mirroring.

    Condition

    Select Object Name Prefix and set the value to examplefolder/.

    Note

    If you configure only one mirroring-based back-to-origin rule for the bucket, you can leave the Object Name Prefix parameter and the File Name Suffix parameter empty. If you configure multiple mirroring-based back-to-origin rules for the bucket, you must set different name prefixes or suffixes to differentiate back-to-origin rules.

    Origin URL

    Select https in the first box, enter www.example.com in the second box, and leave the third box empty.

  6. Click OK.

    The following content shows the access process after the preceding back-to-origin rule is configured:

    1. A requester requests https://examplebucket.oss-cn-hangzhou.aliyuncs.com/examplefolder/example.txt for the first time.

    2. If the examplefolder/example.txt object does not exist in examplebucket, OSS requests the object from the origin server at https://www.example.com/examplefolder/example.txt.

    3. If the object is retrieved from the origin server, OSS writes the example.txt object to the examplefolder directory in examplebucket and then returns the object to the requester. If the object is not retrieved from the origin server, HTTP 404 status code is returned to the requester.

You can use the preceding steps to configure mirroring-based back-to-origin for basic business scenarios. For more information about mirroring-based back-origin examples for specific scenarios, see Configuration examples of mirroring-based back-to-origin.

Use OSS SDKs

The following sample code provides examples on how to configure mirroring-based back-to-origin rules by using OSS SDKs for common programming languages. For more information about how to configure mirroring-based back-to-origin rules by using OSS SDKs for other programming languages, see Overview.

import com.aliyun.oss.ClientException;
import com.aliyun.oss.OSS;
import com.aliyun.oss.common.auth.*;
import com.aliyun.oss.OSSClientBuilder;
import com.aliyun.oss.OSSException;
import com.aliyun.oss.model.RoutingRule;
import com.aliyun.oss.model.SetBucketWebsiteRequest;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

public class Demo {

    public static void main(String[] args) throws Exception {
        // In this example, the endpoint of the China (Hangzhou) region is used. Specify your actual endpoint. 
        String endpoint = "https://oss-cn-hangzhou.aliyuncs.com";
        // Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
        EnvironmentVariableCredentialsProvider credentialsProvider = CredentialsProviderFactory.newEnvironmentVariableCredentialsProvider();
        // Specify the name of the bucket. Example: examplebucket. 
        String bucketName = "examplebucket";

        // Create an OSSClient instance. 
        OSS ossClient = new OSSClientBuilder().build(endpoint, credentialsProvider);

        try {
            SetBucketWebsiteRequest request = new SetBucketWebsiteRequest(bucketName);
            // Specify the operation that you want to perform if the default homepage is specified, the name of the requested object does not end with a forward slash (/), and the object does not exist. 
            //request.setSubDirType(null);
            // Specify whether to redirect the requester to the default homepage in the subdirectory when the subdirectory is accessed. 
            //request.setSupportSubDir(false);

            List<RoutingRule> routingRules = new ArrayList<RoutingRule>();

            RoutingRule rule = new RoutingRule();
            rule.setNumber(1);
            // Specify the prefix that is contained in the object names. Only the objects whose names contain the specified prefix match the rule. 
            rule.getCondition().setKeyPrefixEquals("examplebucket");
            // Specify the HTTP status code. The rule is matched only when the specified object is accessed and HTTP status code 404 is returned. 
            rule.getCondition().setHttpErrorCodeReturnedEquals(404);
            // Specify the redirection type. 
            rule.getRedirect().setRedirectType(RoutingRule.RedirectType.Mirror);
            // Specify the URL of the origin for the mirroring-based back-to-origin rule. Example: https://www.example.com/. 
            rule.getRedirect().setMirrorURL("<yourMirrorURL>");
            //rule.getRedirect().setMirrorRole("AliyunOSSMirrorDefaultRole");
            // Specify whether to include the request parameters when the redirection rule or the mirroring-based back-to-origin rule is executed. 
            rule.getRedirect().setPassQueryString(true);
            // This parameter is used in the same manner as the PassQueryString parameter and is assigned a higher priority level than the PassQueryString parameter. This parameter takes effect only when you set the RedirectType parameter to Mirror. 
            rule.getRedirect().setMirrorPassQueryString(true);
            // Specify the HTTP status code in the response when the request is redirected. This parameter takes effect only when you set the RedirectType parameter to External or AliCDN. 
            //rule.getRedirect().setHttpRedirectCode(302);
            // Specify the domain name that you want to use for redirection. The domain name must comply with the naming conventions for domain names. 
            //rule.getRedirect().setHostName("oss.aliyuncs.com");
            // Specify the protocol that you want to use for redirection. This parameter takes effect only when you set the RedirectType parameter to External or AliCDN. 
            //rule.getRedirect().setProtocol(RoutingRule.Protocol.Https);
            // Specify the string that you want to use to replace the object name when the request is redirected. You can set this parameter to a variable. 
            //rule.getRedirect().setReplaceKeyWith("${key}.jpg");
            // If you set this parameter to true, the prefix of the object name is replaced with the value specified by the ReplaceKeyPrefixWith parameter. 
            rule.getRedirect().setEnableReplacePrefix(true);
            // Specify the string that you want to use to replace the prefix of the object name during redirection. 
            rule.getRedirect().setReplaceKeyPrefixWith("examplebucket");
            // Specify whether to check the MD5 hash of the response body that is returned by the origin. This parameter takes effect only when you set the RedirectType parameter to Mirror. 
            rule.getRedirect().setMirrorCheckMd5(true);

            RoutingRule.MirrorHeaders mirrorHeaders = new RoutingRule.MirrorHeaders();
            // Specify whether to pass through all request headers except the following header to the origin. This parameter takes effect only when you set the RedirectType parameter to Mirror. 
            mirrorHeaders.setPassAll(false);
            List passes = new ArrayList<String>();
            passes.add("cache-control");
            // Specify the headers that you want to pass through to the origin. This parameter takes effect only when you set the RedirectType parameter to Mirror. 
            mirrorHeaders.setPass(passes);
            List removes = new ArrayList<String>();
            removes.add("content-type");
            // Specify the headers that you do not want to pass through to the origin. This parameter takes effect only when you set the RedirectType parameter to Mirror. 
            mirrorHeaders.setRemove(removes);
            List sets = new ArrayList<Map<String, String>>();
            Map header1 = new HashMap<String, String>();
            header1.put("Key", "key1");
            header1.put("Value", "value1");
            Map header2 = new HashMap<String, String>();
            header2.put("Key", "key2");
            header2.put("Value", "value2");
            sets.add(header1);
            sets.add(header2);
            // Specify the headers that you want to pass through to the origin. The specified headers are passed through to the origin regardless of whether the headers are included in the request. 
            mirrorHeaders.setSet(sets);
            // Specify the headers that you want to include in the response when the requested object is obtained from the origin. This parameter takes effect only when you set the RedirectType parameter to Mirror. 
            rule.getRedirect().setMirrorHeaders(mirrorHeaders);

            routingRules.add(rule);
            request.setRoutingRules(routingRules);
            ossClient.setBucketWebsite(request);
        } catch (OSSException oe) {
            System.out.println("Caught an OSSException, which means your request made it to OSS, "
                    + "but was rejected with an error response for some reason.");
            System.out.println("Error Message:" + oe.getErrorMessage());
            System.out.println("Error Code:" + oe.getErrorCode());
            System.out.println("Request ID:" + oe.getRequestId());
            System.out.println("Host ID:" + oe.getHostId());
        } catch (ClientException ce) {
            System.out.println("Caught an ClientException, which means the client encountered "
                    + "a serious internal problem while trying to communicate with OSS, "
                    + "such as not being able to access the network.");
            System.out.println("Error Message:" + ce.getMessage());
        } finally {
            if (ossClient != null) {
                ossClient.shutdown();
            }
        }
    }
}
#-*-coding:utf-8-*-
import oss2
from oss2.models import BucketWebsite, MirrorHeadersSet, RedirectMirrorHeaders, Redirect, RoutingRule, \
    REDIRECT_TYPE_MIRROR, Condition
from oss2.credentials import EnvironmentVariableCredentialsProvider

# Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
auth = oss2.ProviderAuth(EnvironmentVariableCredentialsProvider())
# In this example, the endpoint of the China (Hangzhou) region is used. Specify your actual endpoint. 
# Specify the name of the bucket. Example: examplebucket. 
bucket = oss2.Bucket(auth, 'https://oss-cn-hangzhou.aliyuncs.com', 'examplebucket')

# Enable static website hosting, set the default homepage to index.html, and then set the default 404 page to error.html. 
index_file = 'index.html'
error_file = 'error.html'
# Specify the match conditions. 
condition1 = Condition(key_prefix_equals='examplefolder',
                       http_err_code_return_equals=404)

# Specify the headers that you want to include in the request when you use mirroring-based back-to-origin. 
mirror_headers_set_1 = MirrorHeadersSet("myheader-key5","myheader-value5")
mirror_headers_set_2 = MirrorHeadersSet("myheader-key6","myheader-value6")
set_list = [mirror_headers_set_1, mirror_headers_set_2]
pass_list = ['myheader-key1', 'myheader-key2']
remove_list = ['myheader-key3', 'myheader-key4']
mirror_header = RedirectMirrorHeaders(pass_all=True, pass_list=pass_list, remove_list=remove_list, set_list=set_list)
# Specify the operation to perform after the rule is matched. 
redirect1 = Redirect(redirect_type=REDIRECT_TYPE_MIRROR, mirror_url='https://www.example.com/',
                     mirror_pass_query_string=True, mirror_follow_redirect=True, mirror_check_md5=True, mirror_headers=mirror_header)

rule1 = RoutingRule(rule_num=1, condition=condition1, redirect=redirect1)
website_set = BucketWebsite(index_file, error_file, [rule1])

# Configure mirroring-based back-to-origin. 
bucket.put_bucket_website(website_set)
package main

import (
	"fmt"
	"github.com/aliyun/aliyun-oss-go-sdk/oss"
	"os"
)

func main() {
	// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
	provider, err := oss.NewEnvironmentVariableCredentialsProvider()
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}

	// Create an OSSClient instance. 
	// Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. Specify your actual endpoint. 
	client, err := oss.New("yourEndpoint", "", "", oss.SetCredentialsProvider(&provider))
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}

	// Specify the name of the bucket. Example: examplebucket. 
	bucketName := "examplebucket"

	var indexWebsite = "myindex.html"
	var errorWebsite = "myerror.html"

	btrue := true
	bfalse := false
	// Set the back-to-origin type to mirroring-based back-to-origin. 
	ruleOk := oss.RoutingRule{
		RuleNumber: 1,
		Condition: oss.Condition{
			KeyPrefixEquals: "",
			// Set the back-to-origin condition to HTTP status code 404. 
			HTTPErrorCodeReturnedEquals: 404,
		},
		Redirect: oss.Redirect{
			RedirectType: "Mirror",
			// PassQueryString: &btrue,
			// Specify the back-to-origin URL for a mirroring-based back-to-origin rule. 
			MirrorURL: "http://www.test.com/",
			// MirrorPassQueryString:&btrue,
			// MirrorFollowRedirect:&bfalse,
			// MirrorCheckMd5:&bfalse,
			MirrorHeaders: oss.MirrorHeaders{
				// PassAll:&bfalse,
				// Transmit specific HTTP headers. 
				Pass: []string{"myheader-key1", "myheader-key2"},
				// Prohibit the transmission of specific HTTP headers. 
				Remove: []string{"myheader-key3", "myheader-key4"},
				Set: []oss.MirrorHeaderSet{
					{
						Key:   "myheader-key5",
						Value: "myheader-value5",
					},
				},
			},
		},
	}

	// Set the back-to-origin type to redirection-based back-to-origin. 
	ruleArrOk := []oss.RoutingRule{
		{
			RuleNumber: 2,
			Condition: oss.Condition{
				// Set the back-to-origin condition to HTTP status code 404 and the prefix in object names to abc/. 
				KeyPrefixEquals:             "abc/",
				HTTPErrorCodeReturnedEquals: 404,
				IncludeHeader: []oss.IncludeHeader{
					{
						Key:    "host",
						Equals: "test.oss-cn-beijing-internal.aliyuncs.com",
					},
				},
			},
			Redirect: oss.Redirect{
				RedirectType:     "AliCDN",
				Protocol:         "http",
				HostName:         "www.test.com",
				PassQueryString:  &bfalse,
				ReplaceKeyWith:   "prefix/${key}.suffix",
				HttpRedirectCode: 301,
			},
		},
		// Set the back-to-origin type to mirroring-based back-to-origin. 
		{
			RuleNumber: 3,
			Condition: oss.Condition{
				KeyPrefixEquals:             "",
				HTTPErrorCodeReturnedEquals: 404,
			},
			Redirect: oss.Redirect{
				RedirectType:          "Mirror",
				PassQueryString:       &btrue,
				MirrorURL:             "http://www.test.com/",
				MirrorPassQueryString: &btrue,
				MirrorFollowRedirect:  &bfalse,
				MirrorCheckMd5:        &bfalse,
				MirrorHeaders: oss.MirrorHeaders{
					PassAll: &btrue,
					Pass:    []string{"myheader-key1", "myheader-key2"},
					Remove:  []string{"myheader-key3", "myheader-key4"},
					Set: []oss.MirrorHeaderSet{
						{
							Key:   "myheader-key5",
							Value: "myheader-value5",
						},
					},
				},
			},
		},
	}

	wxmlOne := oss.WebsiteXML{
		IndexDocument: oss.IndexDocument{
			Suffix: indexWebsite,
		},
		ErrorDocument: oss.ErrorDocument{
			Key: errorWebsite,
		},
	}
	wxmlOne.RoutingRules = append(wxmlOne.RoutingRules, ruleOk)
	wxmlOne.RoutingRules = append(wxmlOne.RoutingRules, ruleArrOk...)
	err = client.SetBucketWebsiteDetail(bucketName, wxmlOne)
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
}

     

Use ossutil

You can use ossutil to configure mirroring-based back-to-origin rules. For more information, see Add or modify website-related configurations.

Use the OSS API

If your business requires a high level of customization, you can directly call RESTful APIs. To directly call an API, you must include the signature calculation in your code. For more information, see PutBucketWebsite.

References

You can use the real-time log query feature to query objects uploaded to a bucket by using back-to-origin. For more information, see Real-time log query.