Elasticsearch:Features by version

Version 9.3

New open-source features:

• Agent Builder is generally available (GA). You can interact with your Elasticsearch data in Kibana through conversations, build AI-powered question and answer (Q&A) applications, and use out-of-the-box capabilities such as built-in agents.

• The Elastic Inference Service (EIS) extension supports Jina AI models, expanding the inference model ecosystem.

• DiskBBQ allows you to search quantized vectors directly from disk without loading full vectors into heap memory. It achieves latency under 20 ms with only 100 MB of memory and supports NVIDIA GPU acceleration for vector processing.

• The ACORN filtered vector search algorithm integrates filtering logic into HNSW graph traversal, improving filtered search speed by up to 5 times without affecting accuracy.

• Built on Lucene 10, which further improves index compression rates and inverted index retrieval efficiency.

• LOOKUP JOIN is GA. It supports cross-index joins within the ES|QL query pipeline and extends to multi-field matching, expression calculations, and execution across remote clusters.

• ES|QL query performance for time-series data is significantly optimized, with latency reduced by up to 5 times. New time-series aggregation commands include RATE, *_OVER_TIME, TBUCKET, and TS.

• Inline Stats now supports multi-stage logic, allowing you to perform aggregations mid-pipeline with the stats command while retaining row-level detail data.

• Optimizes garbage collection (GC) for memory fluctuations in high-concurrency write scenarios, which improves system stability.

Alibaba Cloud Elasticsearch 9.3 introduces significant updates. For AI, Agent Builder lets you build Q&A applications in Kibana, and the EIS inference service adds Jina AI model support. For vector search, DiskBBQ and ACORN substantially reduce memory usage and boost filtered search performance, with added GPU acceleration support. For query and analysis, ES|QL gains LOOKUP JOIN for cross-index queries, Inline Stats for multi-stage aggregations, and time-series optimizations that cut latency by up to 5x. The underlying engine upgrades to Lucene 10, improving GC stability for high-concurrency workloads.

For more information about the changes, see What’s new in 9.x.

Version 8.17

New open-source features:

• The dense_vector field introduces Better Binary Quantization (BBQ), a quantization type that compresses vector indexes by 32x and significantly reduces memory usage.

• The Inference API is generally available (GA). For more information, see Inference APIs.

• The Reciprocal Rank Fusion (RRF) feature is GA. For more information, see Reciprocal rank fusion.

• The logsdb index mode is GA. This mode reduces log index storage by approximately 3x. For more information, see Logs data stream.

• Introduces the built-in Elastic Rerank model. For more information, see Elastic Rerank.

• The best_compression codec now uses zstd, reducing storage by about 12% and improving write throughput by 14%.

• ES|QL is optimized with several features, including support for full-text search. For more information, see ES|QL.

The enhanced edition based on Elasticsearch 8.17 enables you to build AI-powered search applications with a built-in model service and supports any external AI model service. Better Binary Quantization (BBQ) reduces memory costs by over 10x.

For more information about the changes, see What’s new in 8.17 and What’s new in 8.16.

Version 8.15

New open-source features:

• Vector index fields are optimized. For more information, see dense-vector.

  • The int8_hnsw type replaces hnsw as the default, with int8 quantization enabled by default.

  • Supports int4 quantization, reducing memory usage by up to 8x.

  • Adds the bit vector type.

• Uses SIMD instructions to accelerate merge performance of int8 quantized indexes on aarch64 by approximately 3x.

• Adds a rerank phase, where the text_similarity_reranker can use a rerank model. For more information, see text-similarity-reranker-retriever.

• Adds the retriever query syntax to support multi-channel recall. For more information, see retriever.

• Adds the semantic_text field type for improved semantic search. For more information, see semantic-text.

• Sparse queries use the sparse_vector syntax instead of text_expansion. For more information, see query-dsl-sparse-vector-query.

• The query rules API is GA. For more information, see query-rules-apis.

• Index Sorting supports nested fields. For more information, see index-modules-index-sorting.

• Adds the efficient logsdb index mode for logging scenarios. For more information, see logs-data-stream.

• Upgrades to Lucene 9.11, improving memory efficiency and query performance. For more information, see apache-lucenetm-9110-available.

For more information about the changes, see What’s new in 8.15 and What’s new in 8.14.

Version 8.13

New open-source features:

• The maximum vector dimension is increased to 4,096. For more information, see 4096 dimension dense vector.

• Vector indexes support scalar quantization, reducing vector index memory usage by nearly 75%. For more information, see scalar-quantization-in-lucene.

• Supports the sparse_vector type for sparse vectors. For more information, see Sparse vector.

• Supports parallel queries on a single shard. For more information, see Query parallelization.

• Vector fields now support the nested type, allowing you to split a document into passages and create vector indexes for each. For more information, see Multiple results from the same doc with nested vectors.

• Adds the Learning To Rank feature, which supports result re-ranking during the rescore phase. For more information, see Learning To Rank.

• Supports the new inference API to integrate with external model services. For more information, see inference APIs.

• Uses SIMD to improve vector query performance. For more information, see Accelerating vector search with SIMD instructions.

For more information about the changes, see What’s new in 8.13.

Version 8.9

New open-source features:

• Supports hybrid ranking of text and vector recall results. For more information, see Reciprocal rank fusion (RRF).

• Increases the maximum vector dimensions to 2,048. For more information, see Increase max number of vector dims to 2048.

• Improves brute-force search performance. For more information, see Improve brute force vector search speed.

• k-NN search supports querying multiple fields simultaneously. For more information, see Allow more than one KNN search clause.

• Provides the built-in ELSER model. For more information, see ELSER inference integration.

• Provides stable support for distributed NLP model scheduling and management. For more information, see Make native inference generally available.

• Improves write performance for documents with primary keys. For more information, see Primary key optimization.

• Improves query performance for constant keyword fields. For more information, see Skip shards when querying constant keyword fields.

• Provides stable support for Time Series Data Streams (TSDS) and the downsampling feature. For more information, see TSDS and Downsample.

• Optimizes source document memory by removing the use of ThreadLocal. For more information, see Remove uses of deprecated LeafReader.

For more information about the changes, see What’s new in 8.9.

Version 8.5

New open-source features:

• Adds vector similarity search based on the HNSW algorithm. For more information, see k-nearest neighbor (kNN) search.

• Adds the Time Series Data Streams (TSDS) feature. For more information, see Time series data stream (TSDS).

• Adds Geo grid queries. For more information, see Geo grid query.

• Simplifies security configurations. For more information, see Start the Elastic Stack with security enabled automatically.

• Improves the Lucene compression algorithm to reduce index size.

• Enhances range query performance.

• Supports the lookup runtime field type. For more information, see lookup-runtime-fields.

• Implements random sampler aggregation queries. For more information, see Random sampler aggregation.

• Reduces heap memory consumption on master and data nodes.

• Removes the _type mapping. However, version 8.x is compatible with requests from version 7.x. For more information about compatibility, see rest-api-compatibility.

• Provides index protection. By default, the elastic user can only read built-in Elasticsearch indexes.

For more information about the changes, see Breaking changes in 8.5.

Version 7.16

New open-source features:

• Supports SQL queries for cross-cluster search.

• The ingest pipeline supports enrich policies of the range type.

• Optimizes the cache to improve query performance.

• Add and remove indexes from a Data Stream.

• Adds cluster UUID and name information to audit logs.

For more information about the changes, see breaking changes in 7.16.

Version 7.10

New open-source features:

• Improves the compression of stored fields to reduce storage costs.

• Enhances Elasticsearch security with Event Query Language (EQL).

• The default value of search.max_buckets is increased from 10,000 to 65,535.

• Adds support for case-insensitive queries. You can enable this by setting the optional case_insensitive parameter to true.

For more information about the changes, see Breaking changes in 7.10.

Version 7.7

New open-source features:

• New indexes now default to one shard instead of five.

• Removes mapping types. You no longer need to specify a type when you define index mappings and templates. For more information, see Removal of mapping types.

• A search request returns a maximum of 10,000 documents by default. If this limit is exceeded, only 10,000 documents are returned. For more information, see track_total_hits 10000 default.

• A single data node can contain a maximum of 1,000 shards by default. You can configure this limit by using the cluster.max_shards_per_node parameter. For more information, see Cluster Shard Limit.

• The total number of scroll contexts is limited to 500 by default. You can configure this limit by using the search.max_open_scroll_context parameter. For more information, see Scroll Search Context.

• The parent circuit breaker now uses real memory (indices.breaker.total.use_real_memory), defaulting to 95% of the JVM heap memory to maximize availability and prevent OutOfMemory errors. For more information, see Circuit Breaker.

• Removes support for the _all field to improve search performance.

• Adds Intervals Queries, which find documents based on the order and proximity of terms.

• When auditing is enabled, audit events are written to the <clustername>_audit.json file on the host's file system. Storing audit events in an index is not supported. For more information, see Enabling audit logging.

For more information about the changes, see Breaking changes in 7.0.

Version 6.x (6.7 and 6.8)

New open-source features:

• An index can have only one type. The _doc type is recommended.

• Starting from version 6.6.0, this version adds index lifecycle management (ILM) to reduce maintenance costs.

• Adds the Rolling up historical data feature to summarize historical data.

• Starting from version 6.3, X-Pack SQL is supported, allowing you to convert SQL statements to DSL queries and reducing the learning curve for DSL.

• Adds more aggregation functions, including Composite, Parent, and Weighted Avg.

For more information about the changes, see Breaking changes in 6.0.

Version 5.x (5.6)

New open-source features:

• An index can have multiple types, and you can define custom types.

• The string field type is deprecated in favor of text or keyword.

• The value of the index mapping parameter is changed from not_analyzed or no to true or false.

• The float data type is used instead of double to reduce storage costs.

• The Java High Level REST Client replaces TransportClient.

For more information about the changes, see Breaking changes in 5.0.