codec-compression is an index compression plug-in developed by Alibaba Cloud Elasticsearch. It supports brotli and zstd compression algorithms and provides a high compression ratio for indexes. This plug-in significantly reduces index storage costs.
Prerequisites
- An Alibaba Cloud Elasticsearch V6.7.0 cluster is created. For more information, see
Create an Alibaba Cloud Elasticsearch cluster.
Notice The codec-compression plug-in is available only in Alibaba Cloud Elasticsearch V6.7.0.
- The codec-compression plug-in is installed. It is automatically installed for new
clusters.
You can check whether the codec-compression plug-in is installed on the Plug-ins page. If the plug-in is not installed, you must manually install it. For more information, see Install and remove a built-in plug-in.
Background information
- Test environment
- Cluster configuration: 3 data nodes (each with 16 vCPUs and 64 GiB of memory) + 2-TiB standard SSD
- Datasets: 74-GiB nyc_taixs of Rally provided by open source Elasticsearch
- Index settings: default settings (You can call the force merge API to perform operations after data is written.)
- Test results
Compression algorithm Index size (GiB) TPS (document/s) LZ4 (default compression algorithm of Elasticsearch) 35.5 202,682 best_compression (DEFLATE) 26.4 181,686 brotli 24.4 182,593 zstd 24.6 181,393 - Test conclusion
When the codec-compression plug-in uses brotli or zstd, it achieves a 45% higher compression ratio but experiences a 10% reduction in write performance compared with when it uses LZ4. However, it achieves an 8% higher compression ratio and maintains the same write performance compared with when it uses best_compression (DEFLATE).