Alibaba Cloud Elasticsearch Vector Engine User Guide - Elasticsearch

The Alibaba Cloud Elasticsearch vector engine handles large-scale vector data. It combines Elasticsearch’s powerful search capabilities with vector similarity computation. Use it for recommendation systems, image retrieval, and natural language processing. This topic explains how to use the Alibaba Cloud Elasticsearch vector engine efficiently and provides best practices. The vector engine is updated regularly. Use the latest version of Alibaba Cloud Elasticsearch to achieve optimal performance, cost efficiency, and user experience.

Prerequisites

An ES instance is created. If you have not created an instance, see Quick start: From creating an instance to retrieving data to create the latest version of an Alibaba Cloud ES 8.x instance.

Note

The vector engine uses large amounts of off-heap memory to cache vector indexes. To choose the appropriate data node specification and count, estimate off-heap memory usage using the memory calculation guidance later in this topic.

Procedure

1. Create an index

Create an index that supports vector data storage. Here is an example index definition:

PUT /my_vector_index
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1
  },
  "mappings": {
    "properties": {
      "my_vector": {
        "type": "dense_vector",
        "dims": 3
      },
      "my_text" : {
        "type" : "keyword"
      }
    }
  }
}

Important

The number_of_shards and number_of_replicas settings depend on your data volume and performance requirements.
Use the dense_vector field type to store vector data. Set the dims parameter to the vector dimension.
The dense_vector field type has many parameters. For details, see Dense vector field type.

2. Import data

Import data into your Elasticsearch vector index using methods such as the Bulk API. Here is an example:

PUT my_vector_index/_doc/1
{
  "my_text" : "text1",
  "my_vector" : [0.5, 10, 6]
}

PUT my_vector_index/_doc/2
{
  "my_text" : "text2",
  "my_vector" : [-0.5, 10, 10]
}

Important

Ensure that the vector dimension matches the dimension defined in the index.

3. Perform vector search

Use Elasticsearch’s vector similarity search to find the most similar documents by specifying a query vector. Here is an example query:

GET my_vector_index/_search
{
  "knn": {
    "field": "my_vector",
    "query_vector": [-5, 9, -12],
    "k": 10,
    "num_candidates": 100
  },
  "fields": [ "my_text" ]
}

Parameter	Content
`k`	(Optional) Number of nearest neighbors to return. This value must be less than or equal to num_candidates. By default, it equals the `size` value.
`num_candidates`	(Optional) The number of nearest neighbor candidates to consider for each shard. This parameter has a significant impact on performance and recall rate. A larger `num_candidates` value increases the recall rate but also has a greater performance impact. This value must be greater than k and cannot exceed 10,000. Elasticsearch collects `num_candidates` results from each shard and merges them to find the top k results. Increasing `num_candidates` tends to improve the accuracy of the final k results. The default value is Math.min(1.5 * k, 10,000).

Note

This section describes the k and num_candidates parameters. In HNSW, num_candidates refers to the ef value of the query, which represents the num_candidates nearest documents collected within each shard, while k is the number of documents Elasticsearch returns in the final results.

Additional vector search features include the following:

The knn query supports filter, the similarity parameter to set the minimum score for matching docs, and nested fields.
It supports querying multiple knn fields at once.
It supports exact knn queries using script.
It supports rescore using script.
For the full list of features, see k-nearest neighbor (knn) search.