codec-compression is an index compression plug-in developed by Alibaba Cloud Elasticsearch. It supports brotli and zstd compression algorithms and provides a higher compression ratio for indexes. This significantly reduces index storage costs.

Prerequisites

You have completed the following operations:
  • An Alibaba Cloud Elasticsearch V6.7.0 cluster is created. For more information, see Create an Alibaba Cloud Elasticsearch cluster.
    Notice The codec-compression plug-in is available only in Alibaba Cloud Elasticsearch V6.7.0.
  • The codec-compression plug-in is installed. By default, this plug-in is installed for new clusters.

    You can check whether the codec-compression plug-in is installed on the Plug-ins page. If the plug-in is not installed, you must manually install it. For more information, see Install and remove a built-in plug-in.

Background information

The codec-compression plug-in supports brotli and zstd compression algorithms. It is suitable for scenarios where a large volume of data needs to be written or the storage costs for indexes are high, such as in logging and time series data analysis. The following content shows a performance test performed on the plug-in:
  • Test environment
    • Cluster configuration: 3 data nodes (each with 16 vCPUs and 64 GiB of memory) + 2-TiB standard SSD
    • Datasets: 74-GiB nyc_taixs of Rally provided by open source Elasticsearch
    • Index settings: default settings. force merge can be performed after data writing.
  • Test results
    Compression algorithm Index size (GiB) TPS (document/s)
    LZ4 (default compression algorithm of Elasticsearch) 35.5 202682
    best_compression (DEFLATE) 26.4 181686
    brotli 24.4 182593
    zstd 24.6 181393
  • Test conclusion

    If the codec-compression plug-in uses brotli and zstd, it achieves a 45% higher compression ratio but experiences a 10% reduction in write performance compared with when it uses LZ4. However, it achieves an 8% higher compression ratio and maintains the same write performance compared with when it uses best_compression (DEFLATE).

Procedure

  1. Log on to the Kibana console of your Elasticsearch cluster.
    For more information, see Log on to the Kibana console.
  2. In the left-side navigation pane, click Dev Tools.
  3. On the Console tab of the page that appears, run the following commands to specify different compression algorithms for an index:
    • brotli compression algorithm 
      PUT index-1
{
    "settings": {
        "index": {
            "codec": "brotli"
        }
    }
}
    • zstd compression algorithm 
      PUT index-1
{
    "settings": {
        "index": {
            "codec": "zstd"
        }
    }
}