Data compression

Sending large volumes of log data over the network increases latency and cost. Simple Log Service supports lz4 compression for select API operations, reducing network traffic and improving throughput without data loss.

Compress request data

The following API operations accept lz4-compressed data in the HTTP request body.

PutLogs (PutLogStoreLogs)
PutWebtracking

To send compressed data:

Add x-log-compresstype: lz4 to the HTTP request header.
Compress the HTTP request body using lz4.
Set x-log-bodyrawsize in the request header to the uncompressed body size.
Set Content-Length in the request header to the compressed body size.

Receive compressed data

The PullLogs API operation can return lz4-compressed data.

To receive compressed data:

Set Accept-Encoding: lz4 in the request header. The server returns lz4-compressed data.
Read x-log-bodyrawsize from the response header to get the uncompressed body size. Pass this value to your decompression function.

Examples

Raw logs

The examples use log-sample.json as sample input. Replace this with your actual log data when calling the API.

{
  "__tags__": {},
  "__topic__": "",
  "__source__": "47.100.XX.XX",
  "__logs__": [
    {
      "__time__": "03/22 08:51:01",
      "content": "*************** RSVP Agent started ***************",
      "method": "main",
      "level": "INFO"
    },
    {
      "__time__": "03/22 08:51:01",
      "content": "Specified configuration file: /u/user10/rsvpd1.conf",
      "method": "locate_configFile",
      "level": "INFO"
    },
    {
      "__time__": "03/22 08:51:01",
      "content": "Using log level 511",
      "method": "main",
      "level": "INFO"
    },
    {
      "__time__": "03/22 08:51:01",
      "content": "Get TCP images rc - EDC8112I Operation not supported on socket",
      "method": "settcpimage",
      "level": "INFO"
    },
    {
      "__time__": "03/22 08:51:01",
      "content": "Associate with TCP/IP image name = TCPCS",
      "method": "settcpimage",
      "level": "INFO"
    },
    {
      "__time__": "03/22 08:51:02",
      "content": "registering process with the system",
      "method": "reg_process",
      "level": "INFO"
    },
    {
      "__time__": "03/22 08:51:02",
      "content": "attempt OS/390 registration",
      "method": "reg_process",
      "level": "INFO"
    },
    {
      "__time__": "03/22 08:51:02",
      "content": "return from registration rc=0",
      "method": "reg_process",
      "level": "INFO"
    }
  ]
}

Test program

The following Python code compresses the sample file and prints the compression ratio:

from lz4 import block
with open('log-sample.json', 'rb') as f:
    data = f.read()
    compressed = block.compress(data, store_size=False) # Compress
    print(f'out/in: {len(compressed)}/{len(data)} Bytes')
    print(f'Compression ratio: {len(compressed)/len(data):.2%}')

Compression result

For log-sample.json, the compression ratio is 39.30%. Actual results vary by content — data with more repetition compresses more efficiently.
```
out/in: 542/1379 Bytes
Compression ratio: 39.30%
```

Sample code

Go example

Install the dependency library
```
go get github.com/pierrec/lz4
```

Sample code

import (
    "fmt"
    "log"

    lz4 "github.com/cloudflare/golz4"
)

func main() {
    data := []byte("hello world, hello golang")
    // Compress
    compressed := make([]byte, lz4.CompressBound(data))
    compressedSize, err := lz4.Compress(data, compressed)
    if err != nil {
        log.Fatal(err)
    }
    compressed = compressed[:compressedSize]

    // Decompress
    bodyRawSize := len(data) // To decompress data from Simple Log Service, read x-log-bodyrawsize from the returned HTTP header.
    decompressed := make([]byte, bodyRawSize)
    err = lz4.Uncompress(compressed, decompressed)
    if err != nil {
        log.Fatal(err)
    }
    decompressed = decompressed[:bodyRawSize]
}

Python example

Install the dependency library
```
python3 -m pip install lz4
```

Sample code

from lz4 import block

data = b'hello world, hello sls'
# Compress
compressed = block.compress(data, store_size=False)

# Decompress
body_raw_size=len(data) # When decompressing data from Simple Log Service, read x-log-bodyrawsize from the returned HTTP header.
decompressed = block.decompress(compressed, uncompressed_size=body_raw_size)

Java example

Add the Maven dependency

<dependency>
  <groupId>net.jpountz.lz4</groupId>
  <artifactId>lz4</artifactId>
  <version>1.3.0</version>
</dependency>

Sample code

package sample;

import net.jpountz.lz4.LZ4Compressor;
import net.jpountz.lz4.LZ4Factory;
import net.jpountz.lz4.LZ4FastDecompressor;

import java.io.File;
import java.io.IOException;
import java.nio.file.Files;

public class Sample {
    public static void main(String[] args) throws IOException {
        byte[] data = "hello world, hello sls".getBytes();
        // Compress
        LZ4Compressor compressor = LZ4Factory.fastestInstance().fastCompressor();
        int maxLen = compressor.maxCompressedLength(data.length);
        byte[] buffer = new byte[maxLen];
        int compressedSize = compressor.compress(data, 0, data.length, buffer, 0, maxLen);
        // Copy the buffer data to compressed.
        byte[] compressed = new byte[compressedSize];
        System.arraycopy(buffer, 0, compressed, 0, compressedSize);

        // Decompress
        int bodyRawSize = data.length; // When decompressing data from Simple Log Service, read x-log-bodyrawsize from the returned HTTP header.
        LZ4FastDecompressor decompressor = LZ4Factory.fastestInstance().fastDecompressor();
        byte[] decompressed = new byte[bodyRawSize];
        decompressor.decompress(compressed, 0, decompressed, 0, bodyRawSize);
    }
}

JavaScript example

Use npm or yarn to install the buffer and lz4 dependency libraries.
```
npm install buffer lz4
```

Sample code

import lz4 from 'lz4'
import { Buffer } from 'buffer'

// Compress
const data = 'hello world, hello sls'
const output = Buffer.alloc(lz4.encodeBound(data.length))
const compressedSize = lz4.encodeBlock(Buffer.from(data), output)
const compressed = Uint8Array.prototype.slice.call(output, 0, compressedSize)

// Decompress
const bodyRawSize = data.length; // When decompressing data from Simple Log Service, read x-log-bodyrawsize from the returned HTTP header.
const decompressed = Buffer.alloc(bodyRawSize)
lz4.decodeBlock(Buffer.from(compressed), decompressed)
const result = decompressed.toString()

C++ example

Copy the lz4 and lib folders from the root directory of the C++ SDK repository to your destination folder. When compiling, add the lz4 library search path (for example, -L./lib) and link to lz4 (for example, -llz4). For more information, see Install the C++ SDK.
```
g++ -o your_program your_program.cpp -std=gnu++11 -llz4 -L./lib/
```

Sample code

#include "lz4/lz4.h"
#include <string>
#include <iostream>
using namespace std;

int main()
{
    string data = "hello sls, hello lz4";
    // Compress
    string compressed;
    compressed.resize(LZ4_compressBound(data.size()));
    int compressed_size = LZ4_compress(data.c_str(), &compressed[0], data.size());
    compressed.resize(compressed_size);

    // Decompress
    string decompressed;
    int bodyRawSize = data.size();
    decompressed.resize(bodyRawSize);
    LZ4_decompress_safe(compressed.c_str(), &decompressed[0], compressed.size(), bodyRawSize);
    cout << decompressed << endl;
    return 0;
}