All Products
Search

This Product

    Elasticsearch:Use the codec-compression plug-in of the beta version

    Document Center

    Elasticsearch:Use the codec-compression plug-in of the beta version

    Last Updated:May 07, 2024

    codec-compression is an index compression plug-in developed by Alibaba Cloud Elasticsearch. It supports Brotli and zstd compression algorithms and provides a high compression ratio for indexes. This plug-in significantly reduces index storage costs.

    Prerequisites

    • An Alibaba Cloud Elasticsearch V6.7.0 cluster is created. For more information, see Create an Alibaba Cloud Elasticsearch cluster.

      Important

      The codec-compression plug-in is available only in Alibaba Cloud Elasticsearch V6.7.0.

    • The codec-compression plug-in is installed. It is automatically installed for new clusters.

      You can check whether the codec-compression plug-in is installed on the Plug-ins page. If the plug-in is not installed, you must manually install it. For more information, see Install and remove a built-in plug-in.

    Background information

    The codec-compression plug-in supports Brotli and zstd compression algorithms. It is suitable for scenarios in which large amounts of data need to be written or the storage costs for indexes are high, such as in logging and time series data analysis. The following description provides information about a performance test performed on the plug-in:

    • Test environment

      • Cluster configuration: 3 data nodes, each of which offers 16 vCPUs, 64 GiB of memory, and one 2-TiB standard SSD

      • Datasets: 74-GiB nyc_taixs of Rally provided by open source Elasticsearch

      • Index settings: default settings (You can call the force merge API to perform operations after data is written.)

    • Test results

      Compression algorithm

      Index size (GiB)

      TPS (document/s)

      LZ4 (default compression algorithm of Elasticsearch)

      35.5

      202682

      best_compression (DEFLATE)

      26.4

      181686

      Brotli

      24.4

      182593

      zstd

      24.6

      181393

    • Test conclusion

      When the codec-compression plug-in uses Brotli or zstd, it achieves a 45% higher compression ratio and maintains the same query performance but causes a 10% reduction in write performance compared with when it uses LZ4. However, it achieves an 8% higher compression ratio and maintains the same write and query performance compared with when it uses best_compression (DEFLATE).

    Procedure

    1. Log on to the Kibana console of your Elasticsearch cluster.

      For more information, see Log on to the Kibana console.

    2. In the left-side navigation pane, click Dev Tools.

    3. On the Console tab of the page that appears, run the following commands to specify different compression algorithms for an index:

      • Brotli compression algorithm 

        PUT index-1
{
    "settings": {
        "index": {
            "codec": "brotli"
        }
    }
}

      • zstd compression algorithm 

        PUT index-1
{
    "settings": {
        "index": {
            "codec": "zstd"
        }
    }
}