OSS提供基于MD5和CRC64的数据校验,确保上传、下载和拷贝文件(Object)过程中的数据完整性。

注意事项

  • 本文以华东1(杭州)外网Endpoint为例。如果您希望通过与OSS同地域的其他阿里云产品访问OSS,请使用内网Endpoint。关于OSS支持的Region与Endpoint的对应关系,请参见访问域名和数据中心
  • 本文以OSS域名新建OSSClient为例。如果您希望通过自定义域名、STS等方式新建OSSClient,请参见新建OSSClient

MD5校验

如果上传文件时设置了Content-MD5,OSS会根据接收的内容计算MD5。OSS计算的MD5值和上传提供的MD5值不一致时,则返回InvalidDigest异常,从而保证数据的完整性。返回InvalidDigest异常后,您需要重新上传文件。

以下代码用于上传文件时进行MD5校验:

import com.aliyun.oss.*;
import com.aliyun.oss.common.utils.BinaryUtil;
import com.aliyun.oss.model.ObjectMetadata;
import java.io.ByteArrayInputStream;

public class Demo {
    public static void main(String[] args) throws Throwable {
        // Endpoint以华东1(杭州)为例,其它Region请按实际情况填写。
        String endpoint = "https://oss-cn-hangzhou.aliyuncs.com";
        // 阿里云账号AccessKey拥有所有API的访问权限,风险很高。强烈建议您创建并使用RAM用户进行API访问或日常运维,请登录RAM控制台创建RAM用户。
        String accessKeyId = "yourAccessKeyId";
        String accessKeySecret = "yourAccessKeySecret";
        // 填写Bucket名称,例如examplebucket。
        String bucketName = "examplebucket";
        // 填写Object的完整路径。Object完整路径中不能包含Bucket名称。
        String objectName = "exampledir/object";

        // 创建OSSClient实例。
        OSS ossClient = new OSSClientBuilder().build(endpoint, accessKeyId, accessKeySecret);

        try {
            // 上传字符串。
            String content = "Hello OSS";

            ObjectMetadata meta = new ObjectMetadata();
            // 设置MD5校验。
            String md5 = BinaryUtil.toBase64String(BinaryUtil.calculateMd5(content.getBytes()));
            meta.setContentMD5(md5);

            ossClient.putObject(bucketName, objectName, new ByteArrayInputStream(content.getBytes()), meta);

        } catch (OSSException oe) {
            System.out.println("Caught an OSSException, which means your request made it to OSS, "
                    + "but was rejected with an error response for some reason.");
            System.out.println("Error Message:" + oe.getErrorMessage());
            System.out.println("Error Code:" + oe.getErrorCode());
            System.out.println("Request ID:" + oe.getRequestId());
            System.out.println("Host ID:" + oe.getHostId());
        } catch (ClientException ce) {
            System.out.println("Caught an ClientException, which means the client encountered "
                    + "a serious internal problem while trying to communicate with OSS, "
                    + "such as not being able to access the network.");
            System.out.println("Error Message:" + ce.getMessage());
        } finally {
            if (ossClient != null) {
                ossClient.shutdown();
            }
        }
    }
}            
说明 putObject、getObject、appendObject、postObject、uploadPart支持MD5校验。

CRC64校验

上传、下载和拷贝文件时默认开启CRC数据校验,确保数据的完整性。

说明
  • putObject、getObject、appendObject、uploadPart支持CRC64校验。上传时默认开启CRC校验,如果客户端计算的CRC值与服务端返回的CRC值不一致, 则会抛出InconsistentException异常。
  • 范围下载不支持CRC64校验。
  • CRC64校验会占用一定的CPU,对上传、下载速度均会有影响。
  • 下载文件时CRC64校验

    以下代码用于下载文件时进行CRC64数据完整性校验:

    import com.aliyun.oss.*;
    import com.aliyun.oss.common.utils.IOUtils;
    import com.aliyun.oss.internal.OSSHeaders;
    import com.aliyun.oss.internal.OSSUtils;
    import com.aliyun.oss.model.GetObjectRequest;
    import com.aliyun.oss.model.OSSObject;
    import java.io.BufferedReader;
    import java.io.InputStreamReader;
    
    public class Demo {
        public static void main(String[] args) throws Throwable {
            // Endpoint以华东1(杭州)为例,其它Region请按实际情况填写。
            String endpoint = "https://oss-cn-hangzhou.aliyuncs.com";
            // 阿里云账号AccessKey拥有所有API的访问权限,风险很高。强烈建议您创建并使用RAM用户进行API访问或日常运维,请登录RAM控制台创建RAM用户。
            String accessKeyId = "yourAccessKeyId";
            String accessKeySecret = "yourAccessKeySecret";
            // 填写Bucket名称,例如examplebucket。
            String bucketName = "examplebucket";
            // 填写Object的完整路径。Object完整路径中不能包含Bucket名称。
            String objectName = "exampledir/object";
    
            // 创建OSSClient实例。
            OSS ossClient = new OSSClientBuilder().build(endpoint, accessKeyId, accessKeySecret);
    
            try {
                // 流式下载。
                GetObjectRequest getObjectRequest = new GetObjectRequest(bucketName, objectName);
                OSSObject ossObject = ossClient.getObject(bucketName, objectName);
    
                // 读取文件内容,只有读取文件内容之后才能获取clientCrc。
                System.out.println("Object content:");
                BufferedReader reader = new BufferedReader(new InputStreamReader(ossObject.getObjectContent()));
                while (true) {
                    String line = reader.readLine();
                    if (line == null) break;
    
                    System.out.println("\n" + line);
                }
                // 数据读取完成后,获取的流必须关闭,否则会造成连接泄漏,导致请求无连接可用,程序无法正常工作。
                reader.close();
    
                // 查看客户端是否开启了CRC校验,默认是开启状态。
                Boolean isCrcCheckEnabled = ((OSSClient)ossClient).getClientConfiguration().isCrcCheckEnabled();
                // 查看是否是范围下载请求。范围下载方式不支持CRC校验。
                Boolean isRangGetRequest = getObjectRequest.getHeaders().get(OSSHeaders.RANGE) != null;
    
                // 校验CRC,且只有读取文件内容之后才能获取clientCRC。
                if (isCrcCheckEnabled && !isRangGetRequest) {
                    Long clientCRC = IOUtils.getCRCValue(ossObject.getObjectContent());
                    OSSUtils.checkChecksum(clientCRC, ossObject.getServerCRC(), ossObject.getRequestId());
                }
            } catch (OSSException oe) {
                System.out.println("Caught an OSSException, which means your request made it to OSS, "
                        + "but was rejected with an error response for some reason.");
                System.out.println("Error Message:" + oe.getErrorMessage());
                System.out.println("Error Code:" + oe.getErrorCode());
                System.out.println("Request ID:" + oe.getRequestId());
                System.out.println("Host ID:" + oe.getHostId());
            } catch (ClientException ce) {
                System.out.println("Caught an ClientException, which means the client encountered "
                        + "a serious internal problem while trying to communicate with OSS, "
                        + "such as not being able to access the network.");
                System.out.println("Error Message:" + ce.getMessage());
            } finally {
                if (ossClient != null) {
                    ossClient.shutdown();
                }
            }
        }
    }
  • 追加上传时CRC64校验

    以下代码用于追加上传时进行CRC64数据完整性校验:

    import com.aliyun.oss.ClientException;
    import com.aliyun.oss.OSS;
    import com.aliyun.oss.OSSClientBuilder;
    import com.aliyun.oss.OSSException;
    import com.aliyun.oss.model.*;
    import java.io.ByteArrayInputStream;
    
    public class Demo {
        public static void main(String[] args) throws Exception {
            // Endpoint以华东1(杭州)为例,其它Region请按实际情况填写。
            String endpoint = "https://oss-cn-hangzhou.aliyuncs.com";
            // 阿里云账号AccessKey拥有所有API的访问权限,风险很高。强烈建议您创建并使用RAM用户进行API访问或日常运维,请登录RAM控制台创建RAM用户。
            String accessKeyId = "yourAccessKeyId";
            String accessKeySecret = "yourAccessKeySecret";
            // 填写Bucket名称,例如examplebucket。
            String bucketName = "examplebucket";
            // 填写Object完整路径,例如exampleobject.txt。Object完整路径中不能包含Bucket名称。
            String objectName = "exampleobject.txt";
            // 填写第一次追加内容,例如Hello。
            String firstAppendContent = "Hello";
            // 填写第二次追加内容,例如World。
            String secondAppendContent = "World";
    
            // 创建OSSClient实例。
            OSS ossClient = new OSSClientBuilder().build(endpoint, accessKeyId, accessKeySecret);
    
            try {
                // 第一次追加。
                AppendObjectRequest appendObjectRequest = new AppendObjectRequest(bucketName, objectName, new ByteArrayInputStream(firstAppendContent.getBytes()));
                appendObjectRequest.setPosition(0L);
                // 初始化CRC。初始化CRC之后,SDK内部默认会对上传结果进行CRC校验。
                appendObjectRequest.setInitCRC(0L);
                AppendObjectResult appendObjectResult = ossClient.appendObject(appendObjectRequest);
    
                // 第二次追加。
                appendObjectRequest = new AppendObjectRequest(bucketName, objectName, new ByteArrayInputStream(secondAppendContent.getBytes()));
                appendObjectRequest.setPosition(appendObjectResult.getNextPosition());
                // 初始化CRC设置为已上传数据的CRC。初始化CRC之后,SDK内部默认会对上传结果进行CRC校验。
                appendObjectRequest.setInitCRC(appendObjectResult.getClientCRC());
                ossClient.appendObject(appendObjectRequest);
            } catch (OSSException oe) {
                System.out.println("Caught an OSSException, which means your request made it to OSS, "
                        + "but was rejected with an error response for some reason.");
                System.out.println("Error Message:" + oe.getErrorMessage());
                System.out.println("Error Code:" + oe.getErrorCode());
                System.out.println("Request ID:" + oe.getRequestId());
                System.out.println("Host ID:" + oe.getHostId());
            } catch (ClientException ce) {
                System.out.println("Caught an ClientException, which means the client encountered "
                        + "a serious internal problem while trying to communicate with OSS, "
                        + "such as not being able to access the network.");
                System.out.println("Error Message:" + ce.getMessage());
            } finally {
                if (ossClient != null) {
                    ossClient.shutdown();
                }
            }
        }
    }

相关文档

关于数据校验的完整示例代码,请参见GitHub示例