Elasticsearch - AliES V1.7.0 Released, Significantly Improving Query Performance in Log-related Scenarios
Sep 20 2022Elasticsearch
Target customers: large-sized enterprises that have complex IT systems and require high log query performance and intelligent O&M. Version released: AliES V1.7.0 is released. It supports the analytic-search plug-in, which significantly improves query performance in log-related scenarios. New features: Index merging policies and date histogram aggregation policies are optimized. This improves the performance of unconditional or single-condition queries by more than six times in log query scenarios, such as queries performed on the Discover tab of the Kibana console. In scenarios where more than 1 TB data is added every day, the time to complete a query is reduced from minutes to 5 seconds or even less. Concurrent queries are optimized. For concurrent queries, concurrent data recall is supported. This improves resource utilization and reduces the average time required for data recall in log-related scenarios by 50%. Read-only small segments are continuously merged before forced merging. This improves query performance by 20%. Optimizations: The lightweight compression algorithm LZ4 is used to transmit write requests between client nodes and data nodes. This reduces the network bandwidth overheads of nodes by 30%. Forced merging can be performed in parallel for shards. This reduces the duration of forced merging. Large data blocks in raw text can be compressed, and parameters for the zstd compression algorithm are optimized. This reduces the size of raw text by 8%. In addition, the Patched Frame of Reference (PFOR) method is supported for Lucene postings. This reduces the size of an index by 3%.