Check data transmission integrity by using 64-bit CRC

Background

An error may occur when data is transmitted between the client and the server. Currently, OSS can return the 64-bit CRC value for an object uploaded in any mode. To check the data integrity, the client can compare the 64-bit CRC value with the locally calculated value.

• OSS calculates 64-bit CRC value for newly uploaded object, stores the result as metadata of the object, and then adds the x-oss-hash-crc64ecma header to the returned response header, indicating its 64-bit CRC value. This 64-bit CRC is calculated according to ECMA-182 Standard.
• For the object that already exists on OSS before the 64-bit CRC goes live, OSS does not calculate its 64-bit CRC value. Therefore, its 64-bit CRC value is not returned when such object is obtained.

Operation instructions

Python example

An example of complete Python code is as follows. It shows how to check data transmission integrity based on the 64-bit CRC value.

1. Calculate the 64-bit CRC value.

   import oss2
   from oss2.models import PartInfo
   import os
   import crcmod
   import random
   import string
   do_crc64 = crcmod.mkCrcFun(0x142F0E1EBA9EA3693L, initCrc=0L, xorOut=0xffffffffffffffffL, rev=True)
   def check_crc64(local_crc64, oss_crc64, msg="check crc64"):
   if local_crc64 ! = oss_crc64:
   print "{0} check crc64 failed. local:{1}, oss:{2}.".format(msg, local_crc64, oss_crc64)
   return False
   else:
   print "{0} check crc64 ok.".format(msg)
   return True
   def random_string(length):
   return ''.join(random.choice(string.lowercase) for i in range(length))
   bucket = oss2. Bucket(oss2. Auth(access_key_id, access_key_secret), endpoint, bucket_name)

2. Verify Put Object.

   content = random_string(1024)
    key = 'normal-key'
    result = bucket.put_object(key, content)
    oss_crc64 = result.headers.get('x-oss-hash-crc64ecma', '')
    local_crc64 = str(do_crc64(content))
    check_crc64(local_crc64, oss_crc64, "put object")

3. Verify Get Object.

   result = bucket.get_object(key)
    oss_crc64 = result.headers.get('x-oss-hash-crc64ecma', '')
    local_crc64 = str(do_crc64(result.resp.read()))
    check_crc64(local_crc64, oss_crc64, "get object")

4. Verify Upload Part and Complete.

   part_info_list = []
    key = "multipart-key"
    result = bucket.init_multipart_upload(key)
    upload_id = result.upload_id
    part_1 = random_string(1024 * 1024)
    result = bucket.upload_part(key, upload_id, 1, part_1)
    oss_crc64 = result.headers.get('x-oss-hash-crc64ecma', '')
    local_crc64 = str(do_crc64(part_1))
    #Check whether the uploaded part 1 data is complete
    check_crc64(local_crc64, oss_crc64, "upload_part object 1")
    part_info_list.append(PartInfo(1, result.etag, len(part_1)))
    part_2 = random_string(1024 * 1024)
    result = bucket.upload_part(key, upload_id, 2, part_2)
    oss_crc64 = result.headers.get('x-oss-hash-crc64ecma', '')
    local_crc64 = str(do_crc64(part_2))
    #Check whether the uploaded part 2 data is complete
    check_crc64(local_crc64, oss_crc64, "upload_part object 2")
    part_info_list.append(PartInfo(2, result.etag, len(part_2)))
    result = bucket.complete_multipart_upload(key, upload_id, part_info_list)
    oss_crc64 = result.headers.get('x-oss-hash-crc64ecma', '')
    local_crc64 = str(do_crc64(part_2, do_crc64(part_1)))
    #Check whether the final object on the OSS is consistent with the local file
    check_crc64(local_crc64, oss_crc64, "complete object")

OSS SDK support

Part of the OSS SDK already supports the data validation using crc64 for the upload and download, as shown in the following table: