codec-compression is an index compression plug-in developed by Alibaba Cloud Elasticsearch. It supports brotli and zstd compression algorithms and provides a higher compression ratio for indexes. This significantly reduces index storage costs.

Prerequisites

You have completed the following operations:
  • An Alibaba Cloud Elasticsearch V6.7.0 cluster is created. For more information, see Create an Alibaba Cloud Elasticsearch instance.
    Notice The codec-compression plug-in is available only in Alibaba Cloud Elasticsearch V6.7.0.
  • The codec-compression plug-in is installed. By default, this plug-in is installed for new clusters.

    If the cluster is created before the plug-in is released, you must manually install the plug-in. For more information, see Install and remove a built-in plug-in.

Background information

The codec-compression plug-in supports brotli and zstd compression algorithms. It is suitable for scenarios where a large volume of data needs to be written or the storage costs for indexes are high, such as in logging and time series data analysis. A performance test is as follows:
  • Test environment
    • Node configuration: 3 data nodes (each with 16 vCPUs and 64 GiB of memory) + 2-TiB standard SSD
    • Datasets: 74-GiB nyc_taixs of Rally provided by open-source Elasticsearch
    • Index settings: default settings (force merge after data writing)
  • Test result
    Compression algorithm Index size (GiB) TPS (document/s)
    LZ4 (default compression algorithm of Elasticsearch) 35.5 202,682
    best_compression (DEFLATE) 26.4 181,686
    brotli 24.4 182,593
    zstd 24.6 181,393
  • Test conclusion

    When codec-compression uses brotli and zstd, it achieves a 45% higher compression ratio and experiences a write performance loss of 10% compared with when it uses LZ4. However, it achieves an 8% higher compression ratio and maintains the same write performance compared with when it uses best_compression (DEFLATE).

Procedure

  1. Log on to the Kibana console.
    For more information, see Log on to the Kibana console.
  2. In the left-side navigation pane, click Dev Tools.
  3. On the Console tab, run the following commands to specify different compression algorithms for an index:
    • brotli compression algorithm 
      PUT index-1
{
    "settings": {
        "index": {
            "codec": "brotli"
        }
    }
}
    • zstd compression algorithm 
      PUT index-1
{
    "settings": {
        "index": {
            "codec": "zstd"
        }
    }
}