Elasticsearch:Kernel version release notes

Kernel version 1.7.0

Plug-ins

• The aliyun-timestream plug-in is available. Use it to create, modify, query, and delete time series indexes, simplifying time series data management. For more information, see Overview of aliyun-timestream.

• Prometheus Querying Language (PromQL) statements are supported for querying data stored in Elasticsearch. For more information, see Integrate Elasticsearch with Prometheus and Grafana based on aliyun-timestream to implement integrated monitoring.

Kernel version 1.12.0

Search

• The analysis-dynamic-synonym plug-in is available.

• Primary shard balancing is supported.

• Parameter value lengths in wildcard and prefix queries are now limited.

• Complex queries—including terms and prefix queries on keyword fields—are optimized using doc_values. Query performance improves by up to 80% in low-hit-ratio scenarios.

• Numeric term and terms queries are optimized using doc_values. Query performance improves by up to 80% in low-hit-ratio scenarios.

• BKD-tree term and terms query performance is optimized by 30% using a lazy loading strategy.

Bug fixes

• Task management at the storage layer is improved to resolve an issue where RPC-based communication occasionally stalled.

• The data replication process is improved to prevent the "fail engine" error on replica nodes.

• The replica shard promotion process is improved to prevent index inconsistency between primary and replica shards.

Kernel version 1.10.0

Store/Snapshot

• LuceneVerifyIndexOutput is optimized to improve index restoration speed. For details, see ES pull #96975.

Cluster coordination

• ClusterState is no longer referenced by persistent tasks. In large-scale clusters, dedicated master nodes can accumulate high memory usage. To prevent leader election timeouts in those environments, the default value of cluster.election.initial_timeout is changed from 100 milliseconds to 1 second. For details, see ES pull #90724.

Search

• End-to-end query timeout is added to control overall query duration. When a timeout occurs, partial results are returned instead of failing the request.

• Additional fields are added to access logs.

Bug fixes

• Fixed an issue where the DV update index file referenced by Lucene Merge was deleted by concurrent flush operations. For details, see Lucene pull #13017.

Kernel version 1.9.0

Search

The concurrent query framework is reconstructed for Kernel-enhanced Edition clusters, with the following improvements:

• JVM heap memory is reused, reducing garbage collection (GC) overhead and improving resource utilization.

• Fetch phase duration for raw text retrieval is reduced. With size set to 10,000, the fetch phase is up to 6–10x faster and the overall query duration is reduced by 50%.

• The following aggregation types are now supported in concurrent queries: percentile, percentile ranks, sampler, diversified sampler, significant text, geo_distance, geohash_grid, geotile_grid, geo_bounds, geo_centroid, and scripted_metric aggregations.

• Fields including traceId and a query duration field are added to end-to-end access logs. Use traceId to trace complete query execution across nodes.

• Custom index structure and mapping parsing for raw text are optimized, doubling write performance for raw text.

Caching

For scenarios with few primary queries but a large number of subqueries, caching was not applied to subqueries. To enable caching in these scenarios, run the following API call:

PUT _cluster/settings
{
  "persistent": {
    "search.query_cache_get_wait_lock_enable": "true",
    "search.query_cache_skip_factor": "200000000"
  }
}

k-NN

• Data inconsistency between primary and replica shards in k-NN query scenarios is resolved.

Bug fixes

• Fixed an issue where running GET _cat/node failed after a shard on a node was migrated during a blue-green update.

Kernel version 1.8.0

Plug-ins

The aliyun-timestream plug-in is available for Elasticsearch V7.10.0. It enhances storage and query performance for time series data and supports:

• Creating, modifying, querying, and deleting time series indexes

• Executing PromQL statements to query data stored in Elasticsearch

• Writing data to time series indexes using the InfluxDB line protocol

For more information, see Overview of aliyun-timestream, Integrate Elasticsearch with Prometheus and Grafana based on aliyun-timestream to implement integrated monitoring, and Integrate aliyun-timestream with the InfluxDB line protocol.

Kernel version 1.7.0

Search

The analytic-search plug-in is available. It significantly improves query performance in log scenarios:

• Index merging policies and date histogram aggregation policies are optimized. Unconditional or single-condition queries—such as those on the Kibana Discover page—are more than 6x faster in log query scenarios. In environments ingesting more than 1 TB of data per day, query time drops from minutes to 5 seconds or less.

• Concurrent data recall is supported for concurrent queries, improving resource utilization and reducing average data recall time by 50% in log scenarios.

• Read-only small segments are continuously merged before force merge, improving query performance by 20%.

Performance improvements

• Write requests between client nodes and data nodes are compressed using LZ4. This reduces network bandwidth overhead by 30%.

• Force merge can run in parallel across shards, reducing the total force merge duration.

• Large data blocks in raw text can be compressed, and zstd compression parameters are optimized, reducing raw text size by 8%. The Patched Frame of Reference (PFOR) method is also supported for Lucene postings, reducing index size by an additional 3%.

Bug fixes

• Fixed an issue where the source_reuse_doc_values feature of the aliyun-codec plug-in did not support fields whose names contained periods (.).

Kernel version 1.6.0

Compression

• The source_reuse_doc_values feature is added to the aliyun-codec plug-in to further reduce index sizes and storage costs. For more information, see Use the aliyun-codec plug-in.

Throttling

• The aliyun-qos plug-in is updated to V2.0, adding finer-grained throttling types and parameters. For more information, see Use the aliyun-qos plug-in.

Kernel version 1.5.0

Compression

• The aliyun-codec plug-in is available to enhance kernel-level compression for clusters. For more information, see Use the aliyun-codec plug-in.

Bug fixes

• Fixed a bug related to the search_as_you_type field type. For details, see GitHub issue #65319.

Kernel version 1.4.0

Search

• The aliyun-knn plug-in is updated with improved write performance, script query support, and optimized vector search using hardware-level optimizations.

Throttling

• The aliyun-qos plug-in is optimized for cluster-level throttling. Traffic is automatically distributed across nodes without requiring knowledge of cluster topology or node load, improving cluster usability and stability.

Kernel version 1.3.0

Search

• Slow query isolation is available to limit the impact of anomalous queries on cluster stability.

• The gig plug-in is available. It performs a switchover within seconds when an exception occurs on a cluster node, preventing query jitter caused by anomalous nodes.

  For Elasticsearch V7.10.0 Standard Edition clusters, the gig plug-in is integrated into the aliyun-qos plug-in, which is installed by default.

Replication

• Physical replication is available to improve write performance for indexes with replica shards.

Time series

• The pruning feature is available for time series indexes to improve query performance.

Observability

• Cluster access logs can be viewed. Logs include fields such as Time, Node IP, and Content. Use these logs to troubleshoot issues and analyze requests.

Cluster management

• Dedicated master node scheduling performance is improved by 10x, allowing each dedicated master node to schedule more shards.

Kernel version 1.3.0

Search

• Slow query isolation is available to limit the impact of anomalous queries on cluster stability.

• The gig plug-in is available. It performs a switchover within seconds when an exception occurs on a cluster node, preventing query jitter caused by anomalous nodes.

Kernel version 1.2.0

Replication

• Physical replication is available to improve write performance for indexes with replica shards.

Time series

• The pruning feature is available for time series indexes to improve query performance.

Write performance

• Primary key-based data deduplication during queries is optimized, improving write performance for documents with primary keys by 10%.

Storage

• Finite state transducers (FSTs) that do not occupy JVM heap memory are supported. A single node can store up to 20 TiB of index data.

Kernel version 1.0.2

Observability

• Cluster access logs can be viewed. Logs include fields such as Time, Node IP, and Content. Use these logs to troubleshoot issues and analyze requests.

Kernel version 1.0.1

Circuit breaking

Circuit breaking policies for JVMs are configurable. When JVM heap memory usage reaches 95%, the cluster rejects incoming requests to protect stability. Configure the following parameters:

Kernel version 0.3.0

Cluster management

• Dedicated master node scheduling performance is improved by 10x, allowing each dedicated master node to schedule more shards.

Write performance

• Write performance is improved by 10% and translog flush overhead is reduced.