Same-region replication (SRR) replicates objects across buckets within the same region in an automatic and asynchronous (near real-time) manner. Operations such as creating, overwriting, and deleting objects can be synchronized from a source bucket to destination buckets.
Scenarios
If you cannot transfer data from a country or region due to compliance requirements of local laws and regulations, you can configure SRR rules to transfer and store replicas of the data in the source bucket to multiple destination buckets that are located in the same region as the source bucket. Objects in destination buckets are exact replicas of objects in the source bucket. The objects have the same object names, versioning information, object content, and object metadata such as the creation time, owner, user metadata, and access control lists (ACLs).
Features
SRR supports the following features:
Data replication between buckets in the same region
You can configure SRR rules to replicate data from a source bucket to multiple destination buckets that are located in the same region as the source bucket. By default, you can configure up to 100 SRR rules for a bucket. A bucket can be specified as a source bucket in an SRR rule and a destination bucket in another SRR rule at the same time.
If you want to configure more than 100 SRR rules for a bucket, contact technical support.
Near real-time data replication
You can configure SRR rules to monitor data that is added, removed, or modified in near real time and synchronize these changes to a destination bucket. This ensures data consistency between the source and destination buckets.
Historical data migration
Historical data can be replicated from a source bucket to a destination bucket. This way, two data replicas are individually stored in the source and destination buckets.
Replication progress query
You can view the most recent replication time of the replicated data and the progress of the replication for historical data migration in percentage.
Versioning
SRR ensures eventual consistency between the data in the source and destination buckets for which versioning is enabled. If you configure an SRR rule to replicate only the added and modified data, deletion operations performed on the specified version of an object in the source bucket are not replicated to the destination bucket. However, the delete markers created in the source bucket are replicated to the destination bucket.
Replication of encrypted data
SRR allows you to replicate objects that are not encrypted and objects that are encrypted by using SSE-KMS or SSE-OSS on the Object Storage Service (OSS) server.
Event notification and real-time log query
You can use the following methods to receive notifications when changes are made to objects in source and destination buckets during SRR. These changes include adding, modifying, removing, and overwriting objects.
Set the event type to the following values in the event notification rule: ObjectReplication:ObjectCreated, ObjectReplication:ObjectRemoved, and ObjectReplication:ObjectModified. For more information, see Overview.
Enable real-time log query in the OSS console to obtain the statistics of operations that are performed on objects. For more information, see Query real-time logs.
Usage notes
Billing
After SRR is enabled, you are not charged for the traffic that is generated when you use SRR to replicate objects from a source bucket to a destination bucket in OSS.
Note The traffic generated when you use SRR does not consume the default bandwidth of OSS.
After SRR is enabled, each time an object is replicated, OSS accumulates the number of requests. However, you are not charged for the requests.
When you use SRR to replicate Archive objects from a source bucket to a destination bucket, no data is restored and you are not charged data retrieval fees.
Replication time
In SRR, data is replicated asynchronously (in near real time). The time required to replicate data from the source bucket to the destination bucket ranges from a few minutes to several hours. The replication time varies based on the data size.
Risks of overwriting an object that has the same name
If you configure an SRR rule for two buckets, an object replicated from the source bucket may overwrite an object that has the same name in the destination bucket.
Limits
You can configure SRR only between two unversioned buckets or two versioning-enabled buckets.
You cannot change the versioning status of two buckets for which an SRR rule is configured.
You cannot replicate Cold Archive or Deep Cold Archive objects from a source bucket to a destination bucket.
You cannot replicate Appendable objects from a source bucket to a destination bucket whose storage class is Cold Archive.
Use the OSS console
Log on to the OSS console.
In the left-side navigation pane, click Buckets. On the page that appears, click the name of the bucket for which you want to enable SRR.
In the left-side navigation tree, choose .
On the Same-Region Replication page, click Same-Region Replication.
In the Same-Region Replication panel, configure parameters. The following table describes the parameters.
Parameter | Description |
Destination Bucket | The destination bucket to which you want to replicate data. |
Objects to Replicate | The objects that you want to replicate. All Files in Source Bucket: OSS replicates all objects from the source bucket to the destination bucket. Files with Specified Prefix: OSS replicates the objects whose names contain a specified prefix from the source bucket to the destination bucket. You can specify up to 10 prefixes.
|
Object Tagging | The tags of the objects that you want to replicate to the destination bucket. Objects that contain the specified tags are replicated to the destination bucket. Select Configure Rules and add tags in key-value pairs. You can add up to 10 tags. To configure this parameter, make sure that the following conditions are met: Tags are configured for objects. For more information, see Configure object tagging. The source and destination buckets must be located in the China (Hangzhou) region or Australia (Sydney) region. Versioning is enabled for the source and destination buckets. The Replication Policy parameter is set to Add/Change.
|
Replication Policy | Specify the data replication policy. Add/Change: OSS replicates only object creation and modification operations from the source bucket to the destination bucket. Add/Delete/Change: OSS replicates object creation, modification, and deletion operations from the source bucket to the destination bucket.
|
Replicate Historical Data | Specify whether to replicate historical data (data that exists before you enable SRR) from the source bucket to the destination bucket. |
Replicate Objects Encrypted based on KMS | If KMS-based encryption is configured for the source objects or destination bucket, you must select Yes for the Replicate Objects Encrypted based on KMS parameter and configure the following parameters: CMK ID: The customer master key (CMK) that is used to encrypt the destination objects. If you want to use a CMK to encrypt objects, you must create a CMK in the same region as the destination bucket in the Key Management Service (KMS) console. For more information, see Create a CMK. RAM Role Name: The RAM role that is authorized to perform KMS-based encryption on the destination objects. New RAM Role: A new RAM role in the kms-replication-sourceBucketName-destinationBucketName format is created to encrypt the destination objects by using CMKs. AliyunOSSRole: The AliyunOSSRole role is used to perform KMS-based encryption on the destination objects. If the AliyunOSSRole role does not exist, OSS automatically creates the AliyunOSSRole role when you select this option.
Note If you create a RAM role or modify the permissions of an existing RAM role, make sure that you attach the AliyunOSSFullAccess policy to the role. Otherwise, data may fail to be replicated. You can call the HeadObject operation to query the encryption status of source objects and the GetBucketEncryption operation to query the encryption status of destination buckets.
|
Click OK.
An SRR rule cannot be edited or deleted after it is created.
The replication immediately starts after an SRR rule is configured. You can view the replication progress on the Same-Region Replication page.
The time required to replicate data from the source bucket to the destination bucket ranges from several minutes to several hours based on the data size.
Use OSS SDKs
You can only use OSS SDK for Java, OSS SDK for Python, and OSS SDK for Go to configure SRR rules.
Java
import com.aliyun.oss.ClientException;
import com.aliyun.oss.OSS;
import com.aliyun.oss.common.auth.*;
import com.aliyun.oss.OSSClientBuilder;
import com.aliyun.oss.OSSException;
import com.aliyun.oss.model.AddBucketReplicationRequest;
public class Demo {
public static void main(String[] args) throws Exception {
// In this example, the endpoint of the China (Hangzhou) region is used. Specify your actual endpoint.
String endpoint = "https://oss-cn-hangzhou.aliyuncs.com";
// We recommend that you do not save access credentials in the project code. Otherwise, access credentials may be leaked. As a result, the security of all resources in your account is compromised. In this example, access credentials are obtained from environment variables. You need to configure environment variables before you run the sample code.
EnvironmentVariableCredentialsProvider credentialsProvider = CredentialsProviderFactory.newEnvironmentVariableCredentialsProvider();
// Specify the name of the source bucket. Example: examplebucket.
String bucketName = "examplebucket";
// Specify the destination bucket.
String targetBucketName = "yourTargetBucketName";
// Specify the region in which the destination bucket is located. Make sure that the destination bucket is located in the same region as the source bucket.
String targetBucketLocation = "yourTargetBucketLocation";
// Create an OSSClient instance.
OSS ossClient = new OSSClientBuilder().build(endpoint, credentialsProvider);
try {
AddBucketReplicationRequest request = new AddBucketReplicationRequest(bucketName);
request.setTargetBucketName(targetBucketName);
request.setTargetBucketLocation(targetBucketLocation);
// Specify whether to replicate historical data. By default, historical data is replicated. In this example, this parameter is set to false. This indicates that historical data is not replicated.
request.setEnableHistoricalObjectReplication(false);
// Specify the name of the role that you want to authorize OSS to use to replicate data. If you want to use SSE-KMS to encrypt objects in the destination bucket, you must specify a role.
//request.setSyncRole("yourRole");
// Specify whether to replicate objects that are encrypted by using SSE-KMS.
//request.setSseKmsEncryptedObjectsStatus("Enabled");
// Specify the CMK ID used in SSE-KMS encryption. If Status is set to Enabled, you must specify this parameter.
//request.setReplicaKmsKeyID("3542abdd-5821-4fb5-a425-90adca***");
//List prefixes = new ArrayList();
//prefixes.add("image/");
//prefixes.add("video");
//prefixes.add("a");
//prefixes.add("A");
// Specify prefixes that are contained in the names of the objects that you want to replicate. After you specify prefixes, only objects whose names contain the prefixes are replicated to the destination bucket.
//request.setObjectPrefixList(prefixes);
//List actions = new ArrayList();
//actions.add(AddBucketReplicationRequest.ReplicationAction.ALL);
// Specify the operations that can be replicated to the destination bucket. The default value is ALL. This indicates that all operations performed on objects in the source bucket are replicated to the destination bucket.
//request.setReplicationActionList(actions);
ossClient.addBucketReplication(request);
} catch (OSSException oe) {
System.out.println("Caught an OSSException, which means your request made it to OSS, "
+ "but was rejected with an error response for some reason.");
System.out.println("Error Message:" + oe.getErrorMessage());
System.out.println("Error Code:" + oe.getErrorCode());
System.out.println("Request ID:" + oe.getRequestId());
System.out.println("Host ID:" + oe.getHostId());
} catch (ClientException ce) {
System.out.println("Caught an ClientException, which means the client encountered "
+ "a serious internal problem while trying to communicate with OSS, "
+ "such as not being able to access the network.");
System.out.println("Error Message:" + ce.getMessage());
} finally {
if (ossClient != null) {
ossClient.shutdown();
}
}
}
}
Python
# -*- coding: utf-8 -*-
import oss2
from oss2.models import ReplicationRule
# The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. Using these credentials to perform operations in OSS is a high-risk operation. We recommend that you use a RAM user to call API operations or perform routine O&M. To create a RAM user, log on to the RAM console.
auth = oss2.Auth('yourAccessKeyId', 'yourAccessKeySecret')
# Specify the endpoint of the region in which the source bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com.
# Specify the name of the source bucket. Example: srcexamplebucket.
bucket = oss2.Bucket(auth, 'https://oss-cn-hangzhou.aliyuncs.com', 'srcexamplebucket')
replica_config = ReplicationRule(
# Specify the destination bucket.
target_bucket_name='destexamplebucket',
# Specify the region in which the destination bucket is located. Make sure that the destination bucket is located in the same region as the source bucket.
target_bucket_location='yourTargetBucketLocation'
)
# Specify the prefixes that are contained in the names of the objects that you want to replicate. After you specify prefixes, only objects whose names contain the prefixes are replicated to the destination bucket.
# prefix_list = ['prefix1', 'prefix2']
# Configure the SRR rule.
# replica_config = ReplicationRule(
# prefix_list=prefix_list,
# Specify the operations that can be replicated to the destination bucket. The default value is ALL. This indicates that all operations performed on objects in the source bucket are replicated to the destination bucket.
# action_list=[ReplicationRule.ALL],
# Specify the destination bucket.
# target_bucket_name='destexamplebucket1',
# Specify the region in which the destination bucket is located.
# target_bucket_location='yourTargetBucketLocation',
# Specify whether to replicate historical data. By default, historical data is replicated. In this example, this parameter is set to False. This indicates that historical data is not replicated.
# is_enable_historical_object_replication=False,
# Specify the link that is used to transfer data during data replication.
# target_transfer_type='oss_acc',
# Specify the role that you want to authorize OSS to use to replicate data. If you want to use SSE-KMS to encrypt objects in the destination bucket, you must specify a role.
# sync_role_name='roleNameTest',
# Replicate the objects that are encrypted by using SSE-KMS.
# sse_kms_encrypted_objects_status=ReplicationRule.ENABLED
# Specify the CMK ID used in SSE-KMS encryption. If you want to use SSE-KMS to encrypt the objects that are replicated to the destination bucket, you must specify this parameter.
# replica_kms_keyid='9468da86-3509-4f8d-a61e-6eab1eac****',
#)
# Enable data replication.
bucket.put_bucket_replication(replica_config)
Go
package main
import (
"fmt"
"github.com/aliyun/aliyun-oss-go-sdk/oss"
"os"
)
func HandleError(err error) {
fmt.Println("Error:", err)
os.Exit(-1)
}
// Enable data replication.
func main() {
// Specify the endpoint of the region in which the source bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. Specify your actual endpoint.
// The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. Using these credentials to perform operations in OSS is a high-risk operation. We recommend that you use a RAM user to call API operations or perform routine O&M. To create a RAM user, log on to the RAM console.
client, err := oss.New("yourEndpoint", "yourAccessKeyId", "yourAccessKeySecret")
if err != nil {
HandleError(err)
}
// Specify the name of the source bucket.
srcbucketName := "srcexamplebucket"
// Specify that OSS replicates only the data that is written to the bucket after the replication rule is created and does not replicate historical data in the source bucket.
putXml := `<?xml version="1.0" encoding="UTF-8"?>
<ReplicationConfiguration>
<Rule>
<PrefixSet>
<! -- Specify that objects whose names contain one of the following prefixes are replicated to the destination bucket: prefix_1 and prefix_2. After you specify prefixes, only objects whose names contain the prefixes are replicated to the destination bucket. -->
<! -- To replicate all objects from the source bucket to the destination bucket, do not specify a prefix.
<Prefix>prefix_1</Prefix>
<Prefix>prefix_2</Prefix>
</PrefixSet>
<! -- Specify the operations that can be replicated to the destination bucket. The default value is ALL. This indicates that all operations performed on objects in the source bucket are replicated to the destination bucket. -->
<Action>ALL</Action>
<Destination>
<! -- Specify the destination bucket. -->
<Bucket>destexamplebucket</Bucket>
<! -- Specify the region in which the destination bucket is located. Make sure that the destination bucket is located in the same region as the source bucket. -->
<Location>oss-cn-beijing</Location>
<! -- Specify the link that is used to transfer data during data replication. In this example, this parameter is set to oss_acc. This indicates that the link used to transfer data is accelerated. -->
<TransferType>oss_acc</TransferType>
</Destination>
<! -- Specify whether to replicate historical data. By default, historical data is replicated. In this example, this parameter is set to disabled. This indicates that historical data is not replicated. -->
<HistoricalObjectReplication>disabled</HistoricalObjectReplication>
<! -- Specify the name of the role that you want to authorize OSS to use to replicate data. If you want to use SSE-KMS to encrypt objects in the destination bucket, you must specify a role. -->
<SyncRole>aliyunramrole</SyncRole>
<SourceSelectionCriteria>
<SseKmsEncryptedObjects>
<! -- Specify whether to replicate the objects that are encrypted by using SSE-KMS. -->
<Status>Enabled</Status>
</SseKmsEncryptedObjects>
</SourceSelectionCriteria>
<EncryptionConfiguration>
<! -- Specify the CMK ID used in SSE-KMS. If Status is set to Enabled, you must specify this parameter. -->
<ReplicaKmsKeyID>c4d49f85-ee30-426b-a5ed-95e9139d****</ReplicaKmsKeyID>
</EncryptionConfiguration>
</Rule>
</ReplicationConfiguration>`
err = client.PutBucketReplication(srcbucketName,putXml)
if err != nil {
HandleError(err)
}
}
Use RESTful APIs
If your business requires a high level of customization, you can directly call RESTful APIs. To directly call an API, you must include the signature calculation in your code. For more information, see PutBucketReplication.
FAQ
Does SSR synchronize lifecycle rule-based deletions from the source bucket to the destination bucket?
It depends on whether the replication policy of the SSR rule is Add/Change or Add/Delete/Change.
Add/Change: When objects are deleted from the source bucket based on a lifecycle rule, OSS does not delete their copies from the destination bucket.
Add/Delete/Change: When objects are deleted from the source bucket based on a lifecycle rule, OSS deletes their copies from the destination bucket.
Note In the destination bucket, you may find objects with the same names as those that are deleted from the source bucket based on a lifecycle rule. This does not indicate that the Add/Delete/Change replication policy fails to take effect. The same-name objects in the destination bucket are not real copies of objects that are deleted from the source bucket. They may be separately written to the destination bucket and given the same names as the objects that have been deleted from the source bucket based on a lifecycle rule.
Does SRR support versioning-suspended buckets?
No, SRR does not support versioning-suspended buckets. You can configure SRR only between two unversioned buckets or two versioning-enabled buckets.
Does OSS support chained replication?
No, SRR does not support chained replication. For example, if a data replication rule is configured to replicate data from Bucket A to Bucket B and another data replication rule is configured to replicate data from Bucket B to Bucket C, data in bucket A is only replicated to Bucket B and is not replicated to Bucket C.
If you want to replicate data from Bucket A to Bucket C, you must configure a data replication rule to replicate data from Bucket A to Bucket C.
Why is the replication progress of historical data displayed as 0% for a long period of time?
The replication progress of historical data is not updated in real time. The replication does not start until all objects are scanned. If a large number of objects are stored in your bucket, such as hundreds of millions of objects, several hours is required before the replication progress of historical data is updated. If the replication progress of historical data is not updated, it does not mean that historical data is not replicated to the destination bucket.
You can check for changes in the capacity and inbound and outbound traffic of the destination bucket to check whether the replication has started. For more information about how to query the capacity, inbound traffic, and outbound traffic of a bucket, see View the resource usage of a bucket.
If the destination bucket uses KMS to encrypt data, am I charged for the encryption algorithm operations?
If the destination bucket uses KMS to encrypt data, you are charged for encryption algorithm operations. For more information about the fees, see Billing of KMS.
Can I disable SRR after I enabled it for a bucket?
Yes, you can click Disable Replication in the Actions column on the Same-Region Replication page to disable SRR.
After you disable SRR, the replicated data is stored in the destination bucket. The incremental data in the source bucket is not replicated to the destination bucket.