Elasticsearch:Use the aliyun-codec plug-in

Performance benchmarks

The following results are based on a test index containing 1.2 TiB of cluster log data across 22 primary shards.

Index compression (zstd algorithm, all three storage layers enabled; compared to a cluster where the aliyun-codec plug-in is used but index compression is not enabled):

source_reuse_doc_values (compared to the same cluster without the feature enabled):

Prerequisites

Before you begin, make sure you have:

• An Alibaba Cloud Elasticsearch V7.10.0 cluster. See Create an Alibaba Cloud Elasticsearch cluster.

• The required kernel version for the features you plan to use: To upgrade the kernel version, see Upgrade the version of a cluster.

  • V1.5.0 or later: index compression only

  • V1.6.0 or later: both index compression and source_reuse_doc_values

• The aliyun-codec plug-in installed on your cluster. The plug-in is installed by default on Elasticsearch V7.10.0 clusters. To verify or install it, go to the Plug-ins page in the Elasticsearch console. See Install and remove a built-in plug-in.

Limitations

• Index compression requires Elasticsearch V7.10.0 with kernel V1.5.0 or later. For Elasticsearch V6.7.0 clusters, use the codec-compression plug-in instead. See Use the codec-compression plug-in of the beta version.

• source_reuse_doc_values requires kernel V1.6.0 or later. On clusters that meet this requirement, index compression is enabled by default in aliyun_default_index_template (index.codec is set to true).

• source_reuse_doc_values can only be enabled at index creation time and cannot be disabled after it is enabled.

Enable index compression

The following steps use an Elasticsearch V7.10.0 cluster. Steps may differ for other versions.

1. Log on to the Kibana console of your cluster. See Log on to the Kibana console.

2. In the upper-right corner, click Dev tools.

3. On the Console tab, run a command to enable index compression on your index. The following command enables compression on an existing index named test. By default, the plug-in applies the zstd algorithm to all three storage layers.

   Set a parameter to "" to disable compression for that storage layer.

   PUT test/_settings
   {
     "index.codec": "ali"
   }

   To use different algorithms for specific storage layers, specify them individually. The following example uses zstd for _source and doc_values, and leaves postings uncompressed.

   PUT test/_settings
   {
     "index.codec": "ali",
     "index.doc_value.compression.default": "zstd",
     "index.postings.compression": "",
     "index.source.compression": "zstd"
   }

Enable source_reuse_doc_values

Open source Elasticsearch stores multiple copies of field data: in _source, postings, and doc_values. source_reuse_doc_values reduces index size by pruning the JSON data stored in _source — for applicable fields, Elasticsearch reconstructs source data from doc_values at read time instead of maintaining a duplicate copy.

PUT test
{
  "settings": {
    "index": {
      "ali_codec_service": {
        "source_reuse_doc_values": {
          "enabled": true
        }
      }
    }
  }
}

PUT _cluster/settings
{
  "persistent": {
    "apack.ali_codec_service.source_reuse_doc_values.max_fields": 100
  }
}

PUT _cluster/settings
{
  "persistent": {
    "apack.ali_codec_service.source_reuse_doc_values.strict_max_fields": true
  }
}

PUT test/_settings
{
  "index": {
    "ali_codec_service": {
      "source_reuse_doc_values": {
        "fetch_slice": 2
      }
    }
  }
}

apack.doc_values_fetch:
    size: 8
    queue_size: 1000

Parameter reference

The following table summarizes all parameters for both features.