This topic answers frequently asked questions about data replication for same-account and cross-account scenarios, including same-region and cross-region replication, and provides troubleshooting methods.
Does OSS data replication support custom destination paths or prefixes?
No. Same-region and cross-region data replication do not support custom storage paths for the destination bucket. The path for replicated objects in the destination bucket is the same as their path in the source bucket.
Can I replicate objects based on their last modified time?
No. The data replication feature does not support synchronizing objects based on a specified last modified time range. However, you can use the File last modified feature of Data Online Migration to achieve this. If you specify a time range, Data Online Migration migrates only objects modified within that period.
Why can't I create a data replication rule?
Check if you have the required permissions.
Missing permissions for the RAM user
When a RAM user tries to create a data replication rule in the console, the OK button is unavailable.
Cause: The
oss:PutBucketReplicationpermission is missing.Solution: Grant the
oss:PutBucketReplicationpermission to the RAM user.
When a RAM user creates a data replication rule in the console, custom-created roles are not displayed for selection.
Cause: The
ram:ListRolespermission is missing.Solution: Grant the
ram:ListRolespermission to the RAM user.
For more information, see Grant custom policies to a RAM user.
Missing permissions for the RAM role
Same-account replication
For same-account replication, you must grant the RAM role permissions to perform replication operations between the source and destination buckets. For more information about how to create and authorize a RAM role, see Role types.
Cross-account replication
For cross-account replication from Account A to Account B, you must use Account A to grant the RAM role permissions to replicate data from the source bucket. You must also use Account B to grant the RAM role permissions to write replicated objects to the destination bucket. For more information about how to create and authorize a RAM role, see Grant permissions to a RAM role.
Check if the source and destination buckets have the same versioning status.
The source and destination buckets must both have versioning enabled or both have versioning disabled.
Check if the endpoint or AccessKey information is correct.
When you create a data replication rule by using an SDK or ossutil, check the following configurations:
Check if the endpoints for the regions of the source and destination buckets are correct. For more information, see Regions and endpoints.
Check if the AccessKey used for the replication rule is correct. For more information, see View the AccessKey information of a RAM user.
Why are objects not replicated to the destination bucket?
If objects are not replicated to the destination bucket after you configure a data replication rule, troubleshoot the issue by checking the following possible causes.
Check if the source bucket is configured correctly.
Check if the data replication status is Enabled.
Check if the prefix is correct.
Replicate specific objects: To replicate only objects with a specific prefix, set the Prefix parameter to the desired prefix. For example, if you set the prefix to
log, OSS replicates only objects whose names start withlog, such aslog/date1.txtandlog/date2.txt. OSS does not replicate objects that do not match the prefix, such asdate3.txt.NoteThe prefix cannot end with an asterisk (*), such as
log/*, and cannot include the bucket name.Replicate all objects: To replicate all objects from the source bucket, leave the Prefix parameter empty.
Check if replication of existing objects is enabled. Existing objects are not replicated if this option is disabled.
Check if the objects in the source bucket are replicas from another replication rule.
If an object in a bucket is a replica created by another replication rule, OSS does not replicate that object again. For example, if you configure replication from Bucket A to Bucket B, and from Bucket B to Bucket C, objects replicated from Bucket A to Bucket B are not replicated to Bucket C.
Check if the objects are encrypted by using KMS. If so, you must enable replication for KMS-encrypted objects.
If objects in the source bucket or the destination bucket are encrypted by using a server-side encryption method that uses CMKs managed by KMS (SSE-KMS), you must select Copy and configure the following parameters:
CMK ID: Specify the CMK to encrypt objects in the destination bucket.
You must create a CMK in the KMS console in the same region as the destination bucket. For more information, see Create a CMK.
RAM Role Name: Authorize a RAM role to perform KMS encryption on objects in the destination bucket.
New RAM Role: Creates a new RAM role to perform KMS encryption on the destination objects. The role name is in the format of
kms-replication-<SourceBucketName>-<DestinationBucketName>.AliyunOSSRole: Uses the AliyunOSSRole service-linked role to perform KMS encryption on the destination objects. If you have not created this role, OSS automatically creates it when you select this option.
NoteIf you create a RAM role or modify the permissions of an existing role, make sure that you attach the
AliyunOSSFullAccesspolicy to the role. Otherwise, data replication may fail.
You can call the HeadObject operation to query the encryption status of a source object and the GetBucketEncryption operation to query the encryption status of the destination bucket.
Check if the replication progress is 100%.
Data replication is an asynchronous process that can take from a few minutes to several hours, depending on the data size. If you are replicating a large object, wait for the replication to complete. After the replication progress reaches 100%, check if the object is in the destination bucket.
Why are delete operations not synchronized to the destination bucket?
Cause 1: A retention policy (WORM) is configured for the destination bucket.
Before the retention period of a retention policy ends, no user, including the resource owner, can delete objects from the bucket.
Cause 2: Whether delete operations are replicated depends on the versioning status of the source bucket and the data replication policy.
Source bucket versioning status
Request type
Data replication policy
Result
Unversioned
Initiate a Delete request
Add/Change
Objects are deleted only from the source bucket. Objects in the destination bucket are not deleted.
Add/Delete/Change
Objects are deleted from both the source and destination buckets.
Versioning enabled
Initiate a Delete request without specifying an object version ID
Add/Change
Objects are not deleted from either bucket. Instead, OSS creates a delete marker in the source bucket, and this delete marker is replicated to the destination bucket.
Add/Delete/Change
Initiate a Delete request specifying an object version ID
Add/Change
The specified object version is deleted only from the source bucket. It is not deleted from the destination bucket.
Add/Delete/Change
The specified object version is deleted from both the source and destination buckets.
Verifying data consistency
You can run the following code to verify data consistency between the source and destination buckets after the replication task is complete:
import com.aliyun.oss.OSSClient;
import com.aliyun.oss.common.auth.*;
import com.aliyun.oss.model.*;
import com.aliyun.oss.OSSException;
import com.aliyuncs.exceptions.ClientException;
public class Demo {
public static void main(String[] args) throws ClientException {
// Obtain access credentials from environment variables. Before you run this sample code, make sure that you have configured the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables.
EnvironmentVariableCredentialsProvider credentialsProvider = CredentialsProviderFactory.newEnvironmentVariableCredentialsProvider();
// Replace srcEndpoint with the endpoint of the region where the source bucket is located.
String srcEndpoint = "https://oss-cn-hangzhou.aliyuncs.com";
OSSClient srcClient = new OSSClient(srcEndpoint , credentialsProvider);
// Specify the name of the source bucket.
String srcBucketName = "src-replication-bucket";
// Replace destEndpoint with the endpoint of the region where the destination bucket is located.
String destEndpoint = "https://oss-cn-beijing.aliyuncs.com";
OSSClient destClient = new OSSClient(destEndpoint, credentialsProvider);
// Specify the name of the destination bucket.
String destBucketName = "dest-replication-bucket";
// Call listObjectsV2 if versioning is disabled for the buckets, or call listVersions if versioning is enabled or suspended.
ListObjectsV2Result result;
ListObjectsV2Request request = new ListObjectsV2Request(srcBucketName);
do {
result = srcClient.listObjectsV2(request);
for (OSSObjectSummary summary : result.getObjectSummaries())
{
String objectName = summary.getKey();
ObjectMetadata srcMeta;
try {
// Obtain the metadata of the source object.
srcMeta = srcClient.headObject(srcBucketName, objectName);
} catch (OSSException ossException) {
if (ossException.getErrorCode().equals("NoSuchKey")) {
continue;
} else {
System.out.println("head src-object failed: " + objectName);
}
continue;
}
ObjectMetadata destMeta;
try {
// Obtain the metadata of the destination object.
destMeta = destClient.headObject(destBucketName, objectName);
} catch (OSSException ossException) {
if (ossException.getErrorCode().equals("NoSuchKey")) {
System.out.println("dest-object not exist: " + objectName);
} else {
System.out.println("head dest-object failed: " + objectName);
}
continue;
}
// Check if the CRC-64 values of the source and destination objects are the same.
Long srcCrc = srcMeta.getServerCRC();
String srcMd5 = srcMeta.getContentMD5();
if (srcCrc != null) {
if (destMeta.getServerCRC() != null) {
if (!destMeta.getServerCRC().equals(srcCrc)) {
System.out.println("crc not equal: " + objectName
+ " | srcCrc: " + srcCrc + " | destCrc: " + destMeta.getServerCRC());
}
continue;
}
}
// Check if the MD5 hashes of the source and destination objects are the same.
if (srcMd5!= null) {
if (destMeta.getContentMD5() != null) {
if (!destMeta.getContentMD5().equals(srcMd5)) {
System.out.println("md5 not equal: " + objectName
+ " | srcMd5: " + srcMd5 + " | destMd5: " + destMeta.getContentMD5());
}
continue;
}
}
// Check if the ETags of the source and destination objects are the same.
if (srcMeta.getETag() == null || !srcMeta.getETag().equals(destMeta.getETag())) {
System.out.println("etag not equal: " + objectName
+ " | srcEtag: " + srcMeta.getETag() + " | destEtag: " + destMeta.getETag());
}
}
request.setContinuationToken(result.getNextContinuationToken());
request.setStartAfter(result.getStartAfter());
} while (result.isTruncated());
}
}
Does OSS support chained replication?
No. For example, if you configure a rule to replicate data from Bucket A to Bucket B, and another rule to replicate data from Bucket B to Bucket C, data from Bucket A is replicated only to Bucket B, not to Bucket C.
To replicate data from Bucket A to Bucket C, you must configure a separate data replication rule from Bucket A to Bucket C.
If you enable replication of existing data for a rule from Bucket B to Bucket C, its initial scan may replicate objects that have just arrived in Bucket B from Bucket A.
Bidirectional replication and infinite loops
No. If you configure bidirectional replication between Bucket A and Bucket B, data (both new and existing) replicated from Bucket A to Bucket B is not replicated back to Bucket A. Similarly, data replicated from Bucket B to Bucket A is not replicated back to Bucket B.
Deletion sync with lifecycle rules
If the data replication policy is set to Add/Change, objects deleted by a lifecycle rule in the source bucket are not deleted from the destination bucket.
If the data replication policy is set to Add/Delete/Change, objects deleted by a lifecycle rule in the source bucket are also deleted from the destination bucket.
NoteIf you find an object in the destination bucket that should have been deleted, it does not mean the Add/Delete/Change policy failed. You may have manually written an object with the same name to the destination bucket.
Replication progress stuck at 0%
The progress is not updated in real time
The replication progress for existing objects is updated only after all objects are scanned. If your bucket contains a large number of objects, such as hundreds of millions, the progress update can take several hours. A static progress bar does not necessarily mean that replication has stalled.
You can verify that replication has started by monitoring the storage usage and inbound replication traffic of the destination bucket. For more information, see Query the resource usage of a bucket.
Incorrect permissions policy for the source bucket
When you create a data replication rule, OSS does not check the source bucket's permissions. This means a rule can be created successfully even if the required permissions are missing. However, replication will fail, and the progress will remain at 0%.
To configure the correct permissions, see Permissions for data replication.
Slow data replication
Increase bandwidth
Data replication is an asynchronous, near-real-time process. The time required depends on the amount of data and can range from a few minutes to several hours. Slow replication can be caused by bandwidth limitations. If bandwidth is the issue, contact technical support to request a bandwidth increase.
Enable Replication Time Control (RTC)
With RTC enabled, OSS replicates most objects within seconds and 99.99% of objects within 10 minutes. When you enable RTC, you are charged for the data replication traffic generated by RTC-enabled tasks. For more information, see Use Replication Time Control (RTC).
Tracking replication operations
You can configure event notification rules to monitor changes to objects in the source and destination buckets during replication. Set the event type to ObjectReplication:ObjectCreated, ObjectReplication:ObjectRemoved, or ObjectReplication:ObjectModified to receive notifications about object creation, deletion, updates, and overwrites. For more information, see Use event notifications to monitor object changes in real time.
Replication for buckets with versioning suspended
No. You can enable data replication only between two buckets that are both unversioned or both versioning-enabled.
KMS encryption and API call charges
Yes. Using KMS encryption for the destination bucket incurs charges for KMS API calls. For more information, see KMS 1.0 billing.
Disabling a replication rule
Yes. You can stop data replication by clicking Disable Replication next to the rule.
After you disable replication, the replicated data remains in the destination bucket. OSS no longer replicates incremental data from the source bucket to the destination bucket.
Replication and object order
Replication does not change the modification order of objects.