As the first cloud service provider to launch Elasticsearch 8.9, Alibaba Cloud is not just offering the cutting-edge Elasticsearch Relevance Engine™ (ESRE™); it's blending enhanced AI capabilities with Elasticsearch's native search functionalities to offer users unprecedented innovation and exploration opportunities.
The rapid advancement and widespread application of artificial intelligence have yielded significant outcomes across various industries. As a potent search engine, Alibaba Cloud Elasticsearch has consistently provided enterprises with efficient and precise search services. Now, as the first provider to introduce version 8.9 domestically, Alibaba Cloud offers an integrated search solution that combines the best AI practices with Elastic's native capabilities, opening new frontiers for user creativity and inquiry.
The latest upgrade from version 8.5 to 8.9 introduces key features that notably augment Alibaba Cloud Elasticsearch's capabilities in vector search and hybrid search, significantly enhancing the accuracy and relevance of search results.
1）Support for mixed ranking of text and vector retrieval results through Reciprocal Rank Fusion (RRF).
2）Extended vector dimensionality up to 2048.
3）Improved brute force search performance.
4）Support for KNN queries spanning multiple fields.
5）Inclusion of built-in ELSER models.
6）Reliable support for distributed management of NLP models.
7）And more...
Learn More：
Vector search, a key addition to the 8.x series, transcends traditional keyword-based search. Utilizing the power of machine learning and artificial intelligence, it converts textual content into vector representations. Words within textual data are represented as vectors, and search retrieval is based on calculating the distances between these vectors to determine textual similarity. This translates into more effective and high-fidelity text retrieval. Compared to classic text searches, vector search enhances semantic relationships between words and documents, bringing a notable boost in search relevance. Not only limited to text, this approach extends to images, voice, and other data types, broadening the application spectrum. Most importantly, vector search allows customization of search results to cater to user preferences, offering a truly personalized search experience.
Hybrid Search RRF allows for the comprehensive re-ranking of result sets retrieved in varying ways, culminating in finely-tuned final rankings. By fusing results from BM25 relevance ranking and vector similarity recall through RRF, the overall precision of rankings receives a marked improvement. The superiority of hybrid search compared to single-faceted techniques is clear—it combines multiple search technologies to yield integrated and more accurate results. In terms of adaptability, enterprises can craft search solutions tailored to their unique business needs, enjoying increased flexibility. For empirical evidence of RRF's impact on search result precision and relevance, one can view tests conducted on Alibaba Cloud Elasticsearch with semantic query results optimized via RRF.
The three query types yielded varying results in terms of accuracy—scaled from 'not relevant' to 'completely relevant.' It's evident from the rankings that RRF's ability to synthesize vector and text query results pushes relevant documents - such as "7911557", previously absent from vector results, to the forefront. Simultaneously, RRF spotlighted the importance of documents like "6080460", which the text query originally overlooked, thereby sharpening recall precision.
| RRF Mixed Arrangement Query | Vector search | Text Search | |||
|---|---|---|---|---|---|
| Paragraph ID | accuracy | Paragraph ID | accuracy | Paragraph ID | accuracy |
| 8588222 | 0 | 8588222 | 0 | 7911557 | 3 |
| 8588219 | 3 | 8588219 | 3 | 8588219 | 3 |
| 7911557 | 3 | 6080460 | 3 | 8588222 | 0 |
| 128984 | 3 | 128984 | 3 | 2697752 | 2 |
| 6080460 | 3 | 4254815 | 1 | 128984 | 3 |
| 2697752 | 2 | 6343521 | 1 | 1721142 | 0 |
| 4254815 | 1 | 1020793 | 0 | 8588227 | 0 |
| 1721142 | 0 | 4254811 | 3 | 302210 | 1 |
| 6343521 | 1 | 1959030 | 0 | 2697746 | 2 |
| 8588227 | 0 | 4254813 | 1 | 7350325 | 0 |
With the release of the new version, Alibaba Cloud Elasticsearch reaffirms its commitment to advancing search capabilities, bringing users a smarter and more profound search experience. Moving forward, Alibaba Cloud Elasticsearch continues to innovate, breaking new ground in search technology and expanding possibilities for users.
Learn More：Enhancing Search Accuracy with RRF(Reciprocal Rank Fusion) in Alibaba Cloud Elasticsearch 8.x
Search and Analytics Service Elasticsearch Version: Alibaba Cloud Elasticsearch is a fully managed Elasticsearch cloud service built on the open-source Elasticsearch, supporting out-of-the-box functionality and pay-as-you-go while being 100% compatible with open-source features. Not only does it provide the cloud-ready components of the Elastic Stack, including Elasticsearch, Logstash, Kibana, and Beats, but it also partners with Elastic to offer the free X-Pack (Platinum level advanced features) commercial plugin. This integration includes advanced features such as security, SQL, machine learning, alerting, and monitoring, and is widely used in scenarios such as real-time log analysis, information retrieval, and multi-dimensional data querying and statistical analysis.
For more information about Elasticsearch, please visit https://www.alibabacloud.com/en/product/elasticsearch.
Enhancing Search Accuracy with RRF(Reciprocal Rank Fusion) in Alibaba Cloud Elasticsearch 8.x
Seamless Integration of Third-party NLP Models into Alibaba Cloud Elasticsearch with Elastic Eland
Data Geek - April 11, 2024
Alibaba Clouder - May 11, 2020
Alibaba Cloud MaxCompute - March 24, 2021
Alibaba Cloud Community - July 21, 2023
ApsaraDB - July 8, 2021
Alibaba Clouder - April 1, 2021
Elasticsearch
Alibaba Cloud Elasticsearch is based on the open-source Elasticsearch engine and is designed for search and analytics scenarios.
Learn More
OpenSearch
OpenSearch helps develop intelligent search services.
Learn More
Image Search
An intelligent image search service with product search and generic search features to help users resolve image search requests.
Learn MoreMore Posts by Data Geek
