edit-icon download-icon

OSS Shipper - JSON storage

Last Updated: Mar 21, 2018

This document introduces the configurations about JSON storage for Log Service logs that are shipped to Object Storage Service (OSS). For more information about shipping logs to OSS, see Ship logs to OSS.

The compression types and file addresses of OSS files are as follows.

Compression type File suffix Example of OSS file address
Do Not Compress None oss://oss-shipper-shenzhen/ecs_test/2016/01/26/20/54_1453812893059571256_937
Compress (snappy) .snappy oss://oss-shipper-shenzhen/ecs_test/2016/01/26/20/54_1453812893059571256_937.snappy

Do not compress

An object is combined by multiple logs. Each line of the file is a log in the JSON format. See the following example:

  1. {"__time__":1453809242,"__topic__":"","__source__":"10.170.148.237","ip":"10.200.98.220","time":"26/Jan/2016:19:54:02 +0800","url":"POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=U0UjpekFQOVJW45A&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=pD12XYLmGxKQ%2Bmkd6x7hAgQ7b1c%3D HTTP/1.1","status":"200","user-agent":"aliyun-sdk-java"}

Compress (snappy)

Use Snappy C++ (Snappy.Compress method) to compress the data in none format at the file level. You can obtain the file in none format after decompress the .snappy file.

Decompress the file by using C++ Lib

Download Lib from Snappy official website and use the Snappy.Uncompress method to decompress the .snappy file.

Decompress the file by using Java Lib

Download Lib from xerial snappy-java. Use Snappy.Uncompress or Snappy.SnappyInputStream to decompress the .snappy file. SnappyFramedInputStream is not supported.

  1. <dependency>
  2. <groupId>org.xerial.snappy</groupId>
  3. <artifactId>snappy-java</artifactId>
  4. <version>1.0.4.1</version>
  5. <type>jar</type>
  6. <scope>compile</scope>
  7. </dependency>

Note: Version 1.1.2.1 may not decompress parts of the compressed file because of a bug, which is fixed in version 1.1.2.6.

Snappy.Uncompress

  1. String fileName = "C:\\Downloads\\36_1474212963188600684_4451886.snappy";
  2. RandomAccessFile randomFile = new RandomAccessFile(fileName, "r");
  3. int fileLength = (int) randomFile.length();
  4. randomFile.seek(0);
  5. byte[] bytes = new byte[fileLength];
  6. int byteread = randomFile.read(bytes);
  7. System.out.println(fileLength);
  8. System.out.println(byteread);
  9. byte[] uncompressed = Snappy.uncompress(bytes);
  10. String result = new String(uncompressed, "UTF-8");
  11. System.out.println(result);

Snappy.SnappyInputStream

  1. String fileName = "C:\\Downloads\\36_1474212963188600684_4451886.snappy";
  2. SnappyInputStream sis = new SnappyInputStream(new FileInputStream(fileName));
  3. byte[] buffer = new byte[4096];
  4. int len = 0;
  5. while ((len = sis.read(buffer)) != -1) {
  6. System.out.println(new String(buffer, 0, len));
  7. }

Decompression tool in Linux environment

For Linux environment, a tool used to decompress .snappy file is provided. Click to download the snappy_tool.

  1. ./snappy_tool 03_1453457006548078722_44148.snappy 03_1453457006548078722_44148
  2. compressed.size: 2217186
  3. snappy::Uncompress return: 1
  4. uncompressed.size: 25223660
Thank you! We've received your feedback.