Object Storage Service:Data validation (Python SDK V1)

How it works

Include the Content-MD5 header in a put_object request. OSS calculates the MD5 hash of the received data and compares it against the value in the header. If they don't match, OSS returns InvalidDigest. Upload the object again to retry.

Example

# -*- coding: utf-8 -*-
import oss2
from oss2.credentials import EnvironmentVariableCredentialsProvider

# Load credentials from environment variables.
# Set OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET before running this example.
auth = oss2.ProviderAuthV4(EnvironmentVariableCredentialsProvider())

# Replace with the endpoint for your bucket's region.
# Example: https://oss-cn-hangzhou.aliyuncs.com for China (Hangzhou).
endpoint = "https://oss-cn-hangzhou.aliyuncs.com"

# Required for v4 signatures. Replace with your bucket's region ID.
region = "cn-hangzhou"

# Replace examplebucket with your bucket name.
bucket = oss2.Bucket(auth, endpoint, "examplebucket", region=region)

# Full object path, excluding the bucket name. Example: exampledir/exampleobject.txt
object_name = 'exampledir/exampleobject.txt'

# Read the local file to upload.
with open('/Users/test/Desktop/demo.txt', 'rb') as file:
    content = file.read()

# Compute the Content-MD5 value.
content_md5 = oss2.utils.content_md5(content)
print('content_md5', content_md5)

# Include Content-MD5 in the request headers.
# OSS verifies the hash on receipt and returns InvalidDigest if they don't match.
headers = dict()
headers['Content-MD5'] = content_md5
bucket.put_object(object_name, content, headers=headers)

How it works

CRC-64 validation is enabled by default for uploads. The SDK computes a CRC checksum on the client side and compares it against the checksum returned by OSS. If they don't match, the SDK throws an InconsistentError exception.

For downloads, CRC validation is not automatic — call read() first, then compare client_crc and server_crc manually.

CRC-64 consumes CPU resources and can affect upload and download speeds. Range downloads do not support CRC-64 validation.

CRC-64 for downloads

bucket.get_object() returns a file-like object. CRC is computed only after the stream is fully read, so compare client_crc and server_crc after calling read().

# -*- coding: utf-8 -*-
import oss2
from oss2.credentials import EnvironmentVariableCredentialsProvider

# Load credentials from environment variables.
# Set OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET before running this example.
auth = oss2.ProviderAuthV4(EnvironmentVariableCredentialsProvider())

# Replace with the endpoint for your bucket's region.
endpoint = "https://oss-cn-hangzhou.aliyuncs.com"

# Required for v4 signatures. Replace with your bucket's region ID.
region = "cn-hangzhou"

# Replace examplebucket with your bucket name.
bucket = oss2.Bucket(auth, endpoint, "examplebucket", region=region)

# Full object path, excluding the bucket name.
object_name = 'yourObjectName'

# Check whether CRC validation is enabled.
print('bucket.enable_crc:', bucket.enable_crc)

# Download the object. get_object() returns a file-like object.
object_stream = bucket.get_object(object_name)

# Read the full stream. CRC is computed during read().
print(object_stream.read())

# Compare client and server CRC values after reading.
if object_stream.client_crc != object_stream.server_crc:
    print("The CRC checksum between client and server is inconsistent!")

CRC-64 for append uploads

For append uploads, pass init_crc to enable CRC-64 validation. Set init_crc=0 for the first append, then pass the CRC from the previous result for each subsequent append.

# -*- coding: utf-8 -*-
import oss2
from oss2.credentials import EnvironmentVariableCredentialsProvider

# Load credentials from environment variables.
# Set OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET before running this example.
auth = oss2.ProviderAuthV4(EnvironmentVariableCredentialsProvider())

# Replace with the endpoint for your bucket's region.
endpoint = "https://oss-cn-hangzhou.aliyuncs.com"

# Required for v4 signatures. Replace with your bucket's region ID.
region = "cn-hangzhou"

# Replace examplebucket with your bucket name.
bucket = oss2.Bucket(auth, endpoint, "examplebucket", region=region)

object_name = "yourAppendObjectName"
first_content = "yourFirstContent"
second_content = "yourSecondContent"

# First append: start at position 0, initialize CRC to 0.
# The SDK validates CRC-64 on the returned result automatically.
result = bucket.append_object(object_name, 0, first_content, init_crc=0)

# Second append: use the next_position and crc from the previous result.
result = bucket.append_object(object_name, result.next_position, second_content, init_crc=result.crc)