codec-compression is an index compression plug-in developed by Alibaba Cloud Elasticsearch. It supports brotli and zstd compression algorithms and provides a higher compression ratio for indexes. This significantly reduces index storage costs.
Prerequisites
- An Alibaba Cloud Elasticsearch V6.7.0 cluster is created. For more information, see
Create an Alibaba Cloud Elasticsearch cluster.
Notice The codec-compression plug-in is available only in Alibaba Cloud Elasticsearch V6.7.0.
- The codec-compression plug-in is installed. By default, this plug-in is installed
for new clusters.
You can check whether the codec-compression plug-in is installed on the Plug-ins page. If the plug-in is not installed, you must manually install it. For more information, see Install and remove a built-in plug-in.
Background information
- Test environment
- Cluster configuration: 3 data nodes (each with 16 vCPUs and 64 GiB of memory) + 2-TiB standard SSD
- Datasets: 74-GiB nyc_taixs of Rally provided by open source Elasticsearch
- Index settings: default settings. force merge can be performed after data writing.
- Test results
Compression algorithm Index size (GiB) TPS (document/s) LZ4 (default compression algorithm of Elasticsearch) 35.5 202682 best_compression (DEFLATE) 26.4 181686 brotli 24.4 182593 zstd 24.6 181393 - Test conclusion
If the codec-compression plug-in uses brotli and zstd, it achieves a 45% higher compression ratio but experiences a 10% reduction in write performance compared with when it uses LZ4. However, it achieves an 8% higher compression ratio and maintains the same write performance compared with when it uses best_compression (DEFLATE).