Object Storage Service:Check data integrity by using CRC-64

Usage notes

• The PutObject, AppendObject, PostObject, and MultipartUploadPart operations return a CRC-64 value. After an upload is complete, you can retrieve the CRC-64 value from the server and compare it with the value that you calculated locally.

• When you call the CompleteMultipartUpload operation, the CRC-64 value for the entire object is returned only if all of its parts have a CRC-64 value. If any part lacks a CRC-64 value, the CRC-64 value for the complete object is not returned. For example, if a part was uploaded before this feature was available, it will not have a CRC-64 value.

• The GetObject, HeadObject, and GetObjectMeta operations return the object's CRC-64 value, if available. After a GetObject operation completes, you can retrieve the CRC-64 value from the server and compare it with a locally calculated value.

  Note

  A GET request with a Range header returns the CRC-64 value of the entire object.

• For copy operations, such as CopyObject and UploadPartCopy, the new object or part is not guaranteed to have a CRC-64 value.

Examples

The following Python code provides an example of how to verify data integrity using CRC-64 values.

import oss2
from oss2.credentials import EnvironmentVariableCredentialsProvider
import crcmod
import random
import string
from oss2.models import PartInfo

# Obtain access credentials from environment variables. Before you run this sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured.
auth = oss2.ProviderAuth(EnvironmentVariableCredentialsProvider())
# Specify the endpoint for the region where the bucket is located. For example, if the bucket is in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com.
# Specify the bucket name in yourBucketName.
bucket = oss2.Bucket(auth, 'https://oss-cn-hangzhou.aliyuncs.com', 'yourBucketName')
# Create a CRC-64 check function.
do_crc64 = crcmod.mkCrcFun(0x142F0E1EBA9EA3693, initCrc=0, xorOut=0xffffffffffffffff, rev=True)

# Check the CRC-64 and print the result.
def check_crc64(local_crc64, oss_crc64, msg="check crc64"):
    if local_crc64 != oss_crc64:
        print("{0} check crc64 failed. local:{1}, oss:{2}.".format(msg, local_crc64, oss_crc64))
        return False
    else:
        print("{0} check crc64 ok.".format(msg))
        return True

# Generate a random string of a specified length.
def random_string(length):
    return ''.join(random.choice(string.ascii_lowercase) for i in range(length))

# Generate a random string named content with a length of 1024.
content = random_string(1024)

# Set the object path.
key = 'normal-key'

# Verify PutObject.
result = bucket.put_object(key, content)
oss_crc64 = result.headers.get('x-oss-hash-crc64ecma', '')
local_crc64 = str(do_crc64(oss2.to_bytes(content)))
check_crc64(local_crc64, oss_crc64, "put object")

# Verify GetObject.
result = bucket.get_object(key)
oss_crc64 = result.headers.get('x-oss-hash-crc64ecma', '')
local_crc64 = str(do_crc64(result.resp.read()))
check_crc64(local_crc64, oss_crc64, "get object")

# Verify UploadPart and Complete.
part_info_list = []
key = "multipart-key"
result = bucket.init_multipart_upload(key)
upload_id = result.upload_id
part_1 = random_string(1024 * 1024)
result = bucket.upload_part(key, upload_id, 1, part_1)
oss_crc64 = result.headers.get('x-oss-hash-crc64ecma', '')
local_crc64 = str(do_crc64(oss2.to_bytes(part_1)))
# Check the data integrity of uploaded part 1.
check_crc64(local_crc64, oss_crc64, "upload_part object 1")
part_info_list.append(PartInfo(1, result.etag, len(part_1)))
part_2 = random_string(1024 * 1024)
result = bucket.upload_part(key, upload_id, 2, part_2)
oss_crc64 = result.headers.get('x-oss-hash-crc64ecma', '')
local_crc64 = str(do_crc64(oss2.to_bytes(part_2)))
# Check the data integrity of uploaded part 2.
check_crc64(local_crc64, oss_crc64, "upload_part object 2")
part_info_list.append(PartInfo(2, result.etag, len(part_2)))
result = bucket.complete_multipart_upload(key, upload_id, part_info_list)
oss_crc64 = result.headers.get('x-oss-hash-crc64ecma', '')
local_crc64 = str(do_crc64(oss2.to_bytes(part_2), do_crc64(oss2.to_bytes(part_1))))
# Check whether the final object on OSS is consistent with the local file.
check_crc64(local_crc64, oss_crc64, "complete object")

OSS SDK support

Some OSS SDKs support CRC-64 data integrity checks for uploads and downloads. See the following table for examples.