All Products
Search
Document Center

Object Storage Service:Data replication

Last Updated:Sep 15, 2023

Data replication automatically replicates objects and object operations, such as creation, overwriting, and deletion, from a source bucket to a destination bucket. Object Storage Service (OSS) supports cross-region replication (CRR) and same-region replication (SRR).

Usage notes

  • In this topic, the public endpoint of the China (Hangzhou) region is used. If you want to access OSS by using other Alibaba Cloud services in the same region as OSS, use an internal endpoint. For more information about the regions and endpoints supported by OSS, see Regions and endpoints.

  • In this topic, access credentials are obtained from environment variables. For more information about how to configure access credentials, see Configure access credentials.

  • In this topic, an OSSClient instance is created by using an OSS endpoint. If you want to create an OSSClient instance by using custom domain names or Security Token Service (STS), see Initialization.

  • By default, an Alibaba Cloud account has data replication-related permissions. If you want to replicate data as a RAM user or by using temporary access credentials provided by Security Token Service (STS), you must have the required permissions. For example:

    • To enable data replication, you must have the oss:PutBucketReplication permission.

    • To enable or disable the replication time control (RTC) feature, you must have the oss:PutBucketRtc permission.

    • To query data replication rules, you must have the oss:GetBucketReplication permission.

    • To query available destination regions, you must have the oss:GetBucketReplicationLocation permission.

    • To query the data replication progress, you must have the oss:GetBucketReplicationProgress permission.

    • To disable data replication, you must have the oss:DeleteBucketReplication permission.

Enable data replication

Important

Before you enable data replication, make sure that the source bucket and the destination bucket are unversioned or versioning-enabled.

The following sample code provides an example on how to enable data replication to replicate data from the srcexamplebucket bucket in the China (Hangzhou) region to the destexamplebucket bucket in the China (Beijing) region:

package main

import (
	"encoding/xml"
	"fmt"
	"github.com/aliyun/aliyun-oss-go-sdk/oss"
	"os"
)

func HandleError(err error) {
	fmt.Println("Error:", err)
	os.Exit(-1)
}

// Enable data replication. 
func main() {
	// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
	provider, err := oss.NewEnvironmentVariableCredentialsProvider()
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// Create an OSSClient instance. 
	// Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. Specify your actual endpoint. 
	client, err := oss.New("yourEndpoint", "", "", oss.SetCredentialsProvider(&provider))
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// Specify the name of the source bucket. 
	srcbucketName := "srcexamplebucket"
	// Specify the name of the destination bucket. 
	destBucketName := "destexamplebucket"
	// Specify prefixes prefix_1 and prefix_2 that are contained in the names of the objects that you want to replicate. After you specify the prefixes, only objects whose names contain one of the prefixes are replicated to the destination bucket. 
	// To replicate all objects from the source bucket to the destination bucket, do not specify prefixes. 
	prefix1 := "prefix_1"
	prefix2 := "prefix_2"
	// Specify the customer master key (CMK) ID used in SSE-KMS encryption. If SSE-KMS encryption is enabled, you must specify a key ID. 
	keyId := "c4d49f85-ee30-426b-a5ed-95e9139d"
	// Specify whether to replicate objects that are encrypted by using SSE-KMS. 
	source := "Enabled"
	prefixSet := oss.ReplicationRulePrefix{Prefix: []*string{&prefix1, &prefix2}}
	// Enable the RTC feature. 
	enabled := "enabled"
	reqReplication := oss.PutBucketReplication{
		Rule: []oss.ReplicationRule{
			{
				PrefixSet: &prefixSet,
				// Specify the operations that can be replicated to the destination bucket. The default value is ALL. This indicates that all operations performed on objects in the source bucket are replicated to the destination bucket. 
				Action: "ALL",
				RTC:    &enabled,
				Destination: &oss.ReplicationRuleDestination{
					Bucket: destBucketName,
					// Specify the region in which the destination bucket is located. 
					// If you want to enable CRR, the source and destination buckets must be located in different regions. If you want to enable SRR, the source and destination buckets must be located in the same region. 
					Location: "oss-cn-hangzhou",
					// Specify the link that is used to transfer data during data replication. This example uses oss_acc, which indicates that the link for transfer acceleration is used. 
					TransferType: "oss_acc",
				},
				// Specify whether to replicate historical data. By default, historical data is replicated. In this example, historical data is not replicated. 
				HistoricalObjectReplication: "disabled",
				SyncRole:                    "aliyunramrole",
				EncryptionConfiguration:     &keyId,
				SourceSelectionCriteria:     &source,
			},
		},
	}

	xmlBody, err := xml.Marshal(reqReplication)
	if err != nil {
		HandleError(err)
	}
	err = client.PutBucketReplication(srcbucketName, string(xmlBody))

	if err != nil {
		HandleError(err)
	}

	fmt.Println("Put Bucket Replication Success!")
}

Query data replication rules

The following sample code provides an example on how to query the data replication rules of the bucket named examplebucket:

package main

import (
	"encoding/xml"
	"fmt"
	"github.com/aliyun/aliyun-oss-go-sdk/oss"
	"os"
)

func HandleError(err error) {
	fmt.Println("Error:", err)
	os.Exit(-1)
}

// Query the data replication rules. 
func main() {
	// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
	provider, err := oss.NewEnvironmentVariableCredentialsProvider()
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// Create an OSSClient instance. 
	// Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. Specify your actual endpoint. 
	client, err := oss.New("yourEndpoint", "", "", oss.SetCredentialsProvider(&provider))
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// Specify the name of the source bucket. 
	bucketName := "srcexamplebucket"
	stringData, err := client.GetBucketReplication(bucketName)
	if err != nil {
		HandleError(err)
	}
	var repResult oss.GetBucketReplicationResult
	err = xml.Unmarshal([]byte(stringData), &repResult)
	if err != nil {
		HandleError(err)
	}
	for _, rule := range repResult.Rule {
		fmt.Printf("Rule Id:%s\n", rule.ID)
		if rule.RTC != nil {
			fmt.Printf("Rule RTC:%s\n", *rule.RTC)
		}
		if rule.PrefixSet != nil {
			for _, prefix := range rule.PrefixSet.Prefix {
				fmt.Printf("Rule Prefix:%s\n", *prefix)
			}
		}
		fmt.Printf("Rule Action:%s\n", rule.Action)
		fmt.Printf("Rule Destination Bucket:%s\n", rule.Destination.Bucket)
		fmt.Printf("Rule Destination Location:%s\n", rule.Destination.Location)
		fmt.Printf("Rule Destination TransferType:%s\n", rule.Destination.TransferType)
		fmt.Printf("Rule Status:%s\n", rule.Status)
		fmt.Printf("Rule Historical Object Replication:%s\n", rule.HistoricalObjectReplication)
		if rule.SyncRole != "" {
			fmt.Printf("Rule SyncRole:%s\n", rule.SyncRole)
		}
	}
}

Configure RTC

The following sample code provides an example on how to enable or disable the RTC feature for a CRR rule:

package main

import (
	"fmt"
	"github.com/aliyun/aliyun-oss-go-sdk/oss"
	"os"
)

func HandleError(err error) {
	fmt.Println("Error:", err)
	os.Exit(-1)
}

func main() {
	// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
	provider, err := oss.NewEnvironmentVariableCredentialsProvider()
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// Create an OSSClient instance. 
	// Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. Specify your actual endpoint. 
	client, err := oss.New("yourEndpoint", "", "", oss.SetCredentialsProvider(&provider))
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// Specify the name of the source bucket. 
	bucketName := "srcexamplebucket"
	// Enable RTC. To disable RTC, replace the value "enabled" with "disabled". 
	enabled := "enabled"
	// Specify the ID of the CRR rule. 
	id := "564df6de-7372-46dc-b4eb-10f****"
	rtc := oss.PutBucketRTC{
		RTC: &enabled,
		ID:  id,
	}
	err = client.PutBucketRTC(bucketName, rtc)
	if err != nil {
		HandleError(err)
	}

	fmt.Println("Put Bucket RTC Success!")
}

Query the regions to which data can be replicated

The following sample code provides an example on how to query the regions to which data can be replicated from the examplebucket bucket:

package main

import (
	"encoding/xml"
	"fmt"
	"github.com/aliyun/aliyun-oss-go-sdk/oss"
	"os"
)

func HandleError(err error) {
	fmt.Println("Error:", err)
	os.Exit(-1)
}

func main() {
	// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
	provider, err := oss.NewEnvironmentVariableCredentialsProvider()
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// Create an OSSClient instance. 
	// Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. Specify your actual endpoint. 
	client, err := oss.New("yourEndpoint", "", "", oss.SetCredentialsProvider(&provider))
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// Specify the name of the source bucket. 
	bucketName := "srcexamplebucket"
	// Query the regions to which the data can be replicated. 
	stringData, err := client.GetBucketReplicationLocation(bucketName)
	if err != nil {
		HandleError(err)
	}
	var repLocation oss.GetBucketReplicationLocationResult
	err = xml.Unmarshal([]byte(stringData), &repLocation)
	if err != nil {
		HandleError(err)
	}

	for _, location := range repLocation.Location {
		fmt.Printf("Bucket Replication Location: %s\n", location)
	}

	for _, transferType := range repLocation.LocationTransferType {
		fmt.Printf("Bucket Replication Location Transfer Type Location: %s\n", transferType.Location)
		fmt.Printf("Bucket Replication Location Transfer Type Type: %s\n", transferType.TransferTypes)
	}

	for _, rtcLocation := range repLocation.RTCLocation {
		fmt.Printf("Bucket Replication Location RTC Location: %s\n", rtcLocation)
	}
	fmt.Println("Get Bucket Replication Location Success!")
}

Query the progress of a data replication task

You can query the progress of historical data replication tasks and incremental data replication tasks.

  • The progress of historical data replication tasks is expressed as a percentage. You can query the progress of historical data replication tasks only for buckets for which historical data replication is enabled.

  • The progress of incremental data replication tasks is expressed as a point in time. Data that is stored in the source bucket before the point in time is replicated.

The following sample code provides an example on how to query the progress of the data replication task that has a specific replication rule ID in the examplebucket bucket:

package main

import (
	"encoding/xml"
	"fmt"
	"github.com/aliyun/aliyun-oss-go-sdk/oss"
	"os"
)

func main() {
	// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
	provider, err := oss.NewEnvironmentVariableCredentialsProvider()
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// Create an OSSClient instance. 
	// Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. Specify your actual endpoint. 
	client, err := oss.New("yourEndpoint", "", "", oss.SetCredentialsProvider(&provider))
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// Specify the name of the source bucket. 
	bucketName := "srcexamplebucket"
	ruleId := "564df6de-7372-46dc-b4eb-10f****"
	// Query the progress of the data replication task. 
	stringData, err := client.GetBucketReplicationProgress(bucketName, ruleId)
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	var repProgress oss.GetBucketReplicationProgressResult
	err = xml.Unmarshal([]byte(stringData), &repProgress)
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	for _, repProgressRule := range repProgress.Rule {
		fmt.Printf("Rule Id:%s\n", repProgressRule.ID)
		if repProgressRule.PrefixSet != nil {
			for _, prefix := range repProgressRule.PrefixSet.Prefix {
				fmt.Printf("Rule Prefix:%s\n", *prefix)
			}
		}
		fmt.Printf("Replication Progress Rule Action:%s\n", repProgressRule.Action)
		fmt.Printf("Replication Progress Rule Destination Bucket:%s\n", repProgressRule.Destination.Bucket)
		fmt.Printf("Replication Progress Rule Destination Location:%s\n", repProgressRule.Destination.Location)
		fmt.Printf("Replication Progress Rule Destination TransferType:%s\n", repProgressRule.Destination.TransferType)
		fmt.Printf("Replication Progress Rule Status:%s\n", repProgressRule.Status)
		fmt.Printf("Replication Progress Rule Historical Object Replication:%s\n", repProgressRule.HistoricalObjectReplication)
		if (*repProgressRule.Progress).HistoricalObject != "" {
			fmt.Printf("Replication Progress Rule Progress Historical Object:%s\n", (*repProgressRule.Progress).HistoricalObject)
		}
		fmt.Printf("Replication Progress Rule Progress NewObject:%s\n", (*repProgressRule.Progress).NewObject)
	}
	fmt.Println("Get Bucket Replication Progress Success!")
}

Disable data replication

You can delete the replication rule that is configured for a source bucket to disable data replication for the bucket.

The following sample code provides an example on how to delete the replication rule that has a specific ID from the examplebucket bucket:

package main

import (
	"fmt"
	"github.com/aliyun/aliyun-oss-go-sdk/oss"
	"os"
)

// Disable data replication. 
func main() {
	// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
	provider, err := oss.NewEnvironmentVariableCredentialsProvider()
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// Create an OSSClient instance. 
	// Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. Specify your actual endpoint. 
	client, err := oss.New("yourEndpoint", "", "", oss.SetCredentialsProvider(&provider))
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}

	// Specify the name of the source bucket. 
	srcbucketName := "yourSourceBucket"
	// Specify the replication rule ID that is returned by the GetBucketReplication operation. 
	err = client.DeleteBucketReplication(srcbucketName, "e047ce28-6806-4131-b1da-30142116****")
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
}

References

  • For more information about the API operation that you can call to enable data replication, see PutBucketReplication.

  • For more information about the API operation that you can call to enable or disable the RTC feature for existing CRR rules, see PutBucketRTC.

  • For more information about the API operation that you can call to query data replication rules, see GetBucketReplication.

  • For more information about the API operation that you can call to query regions to which data can be replicated, see GetBucketReplicationLocation.

  • For more information about the API operation that you can call to query the progress of a data replication task, see GetBucketReplicationProgress.

  • For more information about the API operation that you can call to disable data replication, see DeleteBucketReplication.