OSS提供基于MD5和CRC64的数据校验,确保上传、下载和拷贝文件(Object)过程中的数据完整性。
注意事项
- 本文以华东1(杭州)外网Endpoint为例。如果您希望通过与OSS同地域的其他阿里云产品访问OSS,请使用内网Endpoint。关于OSS支持的Region与Endpoint的对应关系,请参见访问域名和数据中心。
- 本文以OSS域名新建OSSClient为例。如果您希望通过自定义域名、STS等方式新建OSSClient,请参见新建OSSClient。
MD5校验
如果上传文件时设置了Content-MD5,OSS会根据接收的内容计算MD5。OSS计算的MD5值和上传提供的MD5值不一致时,则返回InvalidDigest异常,从而保证数据的完整性。返回InvalidDigest异常后,您需要重新上传文件。
以下代码用于上传文件时进行MD5校验:
import com.aliyun.oss.*;
import com.aliyun.oss.common.utils.BinaryUtil;
import com.aliyun.oss.model.ObjectMetadata;
import java.io.ByteArrayInputStream;
public class Demo {
public static void main(String[] args) throws Throwable {
// Endpoint以华东1(杭州)为例,其它Region请按实际情况填写。
String endpoint = "https://oss-cn-hangzhou.aliyuncs.com";
// 阿里云账号AccessKey拥有所有API的访问权限,风险很高。强烈建议您创建并使用RAM用户进行API访问或日常运维,请登录RAM控制台创建RAM用户。
String accessKeyId = "yourAccessKeyId";
String accessKeySecret = "yourAccessKeySecret";
// 填写Bucket名称,例如examplebucket。
String bucketName = "examplebucket";
// 填写Object的完整路径。Object完整路径中不能包含Bucket名称。
String objectName = "exampledir/object";
// 创建OSSClient实例。
OSS ossClient = new OSSClientBuilder().build(endpoint, accessKeyId, accessKeySecret);
try {
// 上传字符串。
String content = "Hello OSS";
ObjectMetadata meta = new ObjectMetadata();
// 设置MD5校验。
String md5 = BinaryUtil.toBase64String(BinaryUtil.calculateMd5(content.getBytes()));
meta.setContentMD5(md5);
ossClient.putObject(bucketName, objectName, new ByteArrayInputStream(content.getBytes()), meta);
} catch (OSSException oe) {
System.out.println("Caught an OSSException, which means your request made it to OSS, "
+ "but was rejected with an error response for some reason.");
System.out.println("Error Message:" + oe.getErrorMessage());
System.out.println("Error Code:" + oe.getErrorCode());
System.out.println("Request ID:" + oe.getRequestId());
System.out.println("Host ID:" + oe.getHostId());
} catch (ClientException ce) {
System.out.println("Caught an ClientException, which means the client encountered "
+ "a serious internal problem while trying to communicate with OSS, "
+ "such as not being able to access the network.");
System.out.println("Error Message:" + ce.getMessage());
} finally {
if (ossClient != null) {
ossClient.shutdown();
}
}
}
}
说明 putObject、getObject、appendObject、postObject、uploadPart支持MD5校验。
CRC64校验
上传、下载和拷贝文件时默认开启CRC数据校验,确保数据的完整性。
说明
- putObject、getObject、appendObject、uploadPart支持CRC64校验。上传时默认开启CRC校验,如果客户端计算的CRC值与服务端返回的CRC值不一致, 则会抛出InconsistentException异常。
- 范围下载不支持CRC64校验。
- CRC64校验会占用一定的CPU,对上传、下载速度均会有影响。
- 下载文件时CRC64校验
以下代码用于下载文件时进行CRC64数据完整性校验:
import com.aliyun.oss.*; import com.aliyun.oss.common.utils.IOUtils; import com.aliyun.oss.internal.OSSHeaders; import com.aliyun.oss.internal.OSSUtils; import com.aliyun.oss.model.GetObjectRequest; import com.aliyun.oss.model.OSSObject; import java.io.BufferedReader; import java.io.InputStreamReader; public class Demo { public static void main(String[] args) throws Throwable { // Endpoint以华东1(杭州)为例,其它Region请按实际情况填写。 String endpoint = "https://oss-cn-hangzhou.aliyuncs.com"; // 阿里云账号AccessKey拥有所有API的访问权限,风险很高。强烈建议您创建并使用RAM用户进行API访问或日常运维,请登录RAM控制台创建RAM用户。 String accessKeyId = "yourAccessKeyId"; String accessKeySecret = "yourAccessKeySecret"; // 填写Bucket名称,例如examplebucket。 String bucketName = "examplebucket"; // 填写Object的完整路径。Object完整路径中不能包含Bucket名称。 String objectName = "exampledir/object"; // 创建OSSClient实例。 OSS ossClient = new OSSClientBuilder().build(endpoint, accessKeyId, accessKeySecret); try { // 流式下载。 GetObjectRequest getObjectRequest = new GetObjectRequest(bucketName, objectName); OSSObject ossObject = ossClient.getObject(bucketName, objectName); // 读取文件内容,只有读取文件内容之后才能获取clientCrc。 System.out.println("Object content:"); BufferedReader reader = new BufferedReader(new InputStreamReader(ossObject.getObjectContent())); while (true) { String line = reader.readLine(); if (line == null) break; System.out.println("\n" + line); } // 数据读取完成后,获取的流必须关闭,否则会造成连接泄漏,导致请求无连接可用,程序无法正常工作。 reader.close(); // 查看客户端是否开启了CRC校验,默认是开启状态。 Boolean isCrcCheckEnabled = ((OSSClient)ossClient).getClientConfiguration().isCrcCheckEnabled(); // 查看是否是范围下载请求。范围下载方式不支持CRC校验。 Boolean isRangGetRequest = getObjectRequest.getHeaders().get(OSSHeaders.RANGE) != null; // 校验CRC,且只有读取文件内容之后才能获取clientCRC。 if (isCrcCheckEnabled && !isRangGetRequest) { Long clientCRC = IOUtils.getCRCValue(ossObject.getObjectContent()); OSSUtils.checkChecksum(clientCRC, ossObject.getServerCRC(), ossObject.getRequestId()); } } catch (OSSException oe) { System.out.println("Caught an OSSException, which means your request made it to OSS, " + "but was rejected with an error response for some reason."); System.out.println("Error Message:" + oe.getErrorMessage()); System.out.println("Error Code:" + oe.getErrorCode()); System.out.println("Request ID:" + oe.getRequestId()); System.out.println("Host ID:" + oe.getHostId()); } catch (ClientException ce) { System.out.println("Caught an ClientException, which means the client encountered " + "a serious internal problem while trying to communicate with OSS, " + "such as not being able to access the network."); System.out.println("Error Message:" + ce.getMessage()); } finally { if (ossClient != null) { ossClient.shutdown(); } } } }
- 追加上传时CRC64校验
以下代码用于追加上传时进行CRC64数据完整性校验:
import com.aliyun.oss.ClientException; import com.aliyun.oss.OSS; import com.aliyun.oss.OSSClientBuilder; import com.aliyun.oss.OSSException; import com.aliyun.oss.model.*; import java.io.ByteArrayInputStream; public class Demo { public static void main(String[] args) throws Exception { // Endpoint以华东1(杭州)为例,其它Region请按实际情况填写。 String endpoint = "https://oss-cn-hangzhou.aliyuncs.com"; // 阿里云账号AccessKey拥有所有API的访问权限,风险很高。强烈建议您创建并使用RAM用户进行API访问或日常运维,请登录RAM控制台创建RAM用户。 String accessKeyId = "yourAccessKeyId"; String accessKeySecret = "yourAccessKeySecret"; // 填写Bucket名称,例如examplebucket。 String bucketName = "examplebucket"; // 填写Object完整路径,例如exampleobject.txt。Object完整路径中不能包含Bucket名称。 String objectName = "exampleobject.txt"; // 填写第一次追加内容,例如Hello。 String firstAppendContent = "Hello"; // 填写第二次追加内容,例如World。 String secondAppendContent = "World"; // 创建OSSClient实例。 OSS ossClient = new OSSClientBuilder().build(endpoint, accessKeyId, accessKeySecret); try { // 第一次追加。 AppendObjectRequest appendObjectRequest = new AppendObjectRequest(bucketName, objectName, new ByteArrayInputStream(firstAppendContent.getBytes())); appendObjectRequest.setPosition(0L); // 初始化CRC。初始化CRC之后,SDK内部默认会对上传结果进行CRC校验。 appendObjectRequest.setInitCRC(0L); AppendObjectResult appendObjectResult = ossClient.appendObject(appendObjectRequest); // 第二次追加。 appendObjectRequest = new AppendObjectRequest(bucketName, objectName, new ByteArrayInputStream(secondAppendContent.getBytes())); appendObjectRequest.setPosition(appendObjectResult.getNextPosition()); // 初始化CRC设置为已上传数据的CRC。初始化CRC之后,SDK内部默认会对上传结果进行CRC校验。 appendObjectRequest.setInitCRC(appendObjectResult.getClientCRC()); ossClient.appendObject(appendObjectRequest); } catch (OSSException oe) { System.out.println("Caught an OSSException, which means your request made it to OSS, " + "but was rejected with an error response for some reason."); System.out.println("Error Message:" + oe.getErrorMessage()); System.out.println("Error Code:" + oe.getErrorCode()); System.out.println("Request ID:" + oe.getRequestId()); System.out.println("Host ID:" + oe.getHostId()); } catch (ClientException ce) { System.out.println("Caught an ClientException, which means the client encountered " + "a serious internal problem while trying to communicate with OSS, " + "such as not being able to access the network."); System.out.println("Error Message:" + ce.getMessage()); } finally { if (ossClient != null) { ossClient.shutdown(); } } } }
相关文档
关于数据校验的完整示例代码,请参见GitHub示例。