Use the kNN search feature of Elasticsearch - Elasticsearch

Alibaba Cloud Elasticsearch V8.0 and later provide the k-nearest neighbor (kNN) search feature. This feature can help you quickly use vector spaces to meet your business requirements in different search scenarios, such as image searching, video fingerprinting, facial recognition, speech recognition, and commodity recommendation. This topic describes how to use the kNN search feature.

Background information

For more information about the kNN search feature of Elasticsearch, see k-nearest neighbor (kNN) search.

Prerequisites

An Alibaba Cloud Elasticsearch V8.X cluster is created. In this example, an Alibaba Cloud Elasticsearch V8.5.1 cluster is created. For more information, see Create an Alibaba Cloud Elasticsearch cluster.
Business data is converted into vector data, and the vector data is stored in fields of the dense_vector type. You can design vectors based on a similarity metric. If the vector of a document is closer to a query vector, the similarity of the vectors is higher.

Precautions

You must store vector data in fields of the dense_vector type. Fields of the dense_vector type do not support aggregation or sorting operations.
Fields of the nested type do not support approximate kNN search.
If you use the kNN search feature in cross-cluster search scenarios, the ccs_minimize_roundtrips parameter is not supported.
By default, kNN search uses the dfs_query_then_fetch search type. When you use the kNN search feature to perform a search, you cannot explicitly configure the search_type parameter.

Search methods that are supported by kNN search

kNN search supports the following search methods: Approximate kNN search and Exact kNN search. The following table describes the differences between the two methods.


Search method	Query interface	In-memory storage	Mapping requirement	Description
Approximate kNN search	Use the search API to configure kNN parameters.	Yes	You must set the index parameter to true for fields of the dense_vector type to enable approximate kNN search. Note The approximate kNN search method is supported in Elasticsearch V8.0 and later. In earlier versions of Elasticsearch, you cannot set the index parameter to true in the mappings for fields of the dense_vector type. If your Elasticsearch cluster is upgraded from a version earlier than V8.0 to V8.5 and you want to perform a kNN search in the cluster, you must make sure that the index that you create contains fields of the dense_vector type. To perform an approximate kNN search, you must also recreate the index and set the index parameter to true in the mappings for the index.	Approximate kNN search sacrifices some indexing speed and accuracy for the sake of lower latency.
Exact kNN search	Use the script_score query with a vector function.	Yes	You can set the index parameter to false or leave the parameter empty for fields of the dense_vector type to improve the search efficiency.	The script_score query scans each matched document to calculate a vector function. This slows down the search speed. You can configure a query clause to limit the number of documents that can be passed to the vector function to improve the search efficiency.

Approximate kNN search

Optimize performance

The approximate kNN search method can help you quickly find k vectors that are nearest to a query vector. The logic of this search method is different from the logic of other search methods. The approximate kNN search method has special requirements for the performance of an Elasticsearch cluster. You can optimize the performance of your Elasticsearch cluster based on the following instructions:

Elasticsearch stores the dense vectors of each segment in Hierarchical Navigable Small World (HNSW) graphs. The construction of HNSW graphs requires a long period of time during the indexing of vector data. We recommend that you increase the timeout period of the client and use a bulk write request to write data.
You can reduce the number of segments of the index or merge all segments of the index into one segment to improve the search efficiency.
You must make sure that the memory space of the data nodes in your Elasticsearch cluster is greater than the total space that is occupied by all vector data and the index structure.
Do not write a large amount of data to or update data in your Elasticsearch cluster during an approximate kNN search.

Create an index

When you create an index in which you want to perform an approximate kNN search, set the index parameter to true and configure the similarity parameter in the mappings for the index. The following code provides a configuration example:

PUT image-index
{
  "mappings": {
    "properties": {
      "image-vector": {
        "type": "dense_vector",
        "dims": 3,
        "index": true,
        "similarity": "l2_norm"
      },
      "title": {
        "type": "text"
      },
      "file-type": {
        "type": "keyword"
      }
    }
  }
}

The following table describes the parameters that are used to define settings related to fields of the dense_vector type. For more information, see dense-vector.


Parameter	Description
type	The data type of the fields that are used to store floating-point numbers. Set this parameter to dense_vector.
dims	The number of dimensions for each vector. If you set the index parameter to true, the value of the dims parameter cannot exceed 1024. If you set the index parameter to false, the value of the dims parameter cannot exceed 2048.
index	Specifies whether to generate a new index for the approximate kNN search. To perform an approximate kNN search, set the index parameter to true. Default value: false.
similarity	The algorithm used to calculate the similarity between a query vector and a document vector. If you set the index parameter to true, you must configure this parameter. Valid values: l2_norm: calculates the Euclidean distance between the vectors. Formula used to calculate the score: `1/(1 + l2_norm(query, vector)^2)`. dot_product: calculates the dot product of two vectors. The calculation of the score depends on the element_type parameter. If you set the element_type parameter to float, both vectors must be normalized to a unit length. Formula used to calculate the score: `(1 + dot_product(query, vector))/2`. If you set the element_type parameter to byte, both vectors must have the same length. If the lengths of the vectors are different, the result about the similarity may be inaccurate. Formula used to calculate the score: `0.5 + (dot_product(query, vector)/(32768 × dims))`. cosine: calculates the cosine similarity between the vectors. The most efficient way to use the cosine algorithm is to normalize both vectors to a unit length to replace the dot_product algorithm. Formula used to calculate the score: `(1 + cosine(query, vector))/2`. Important The cosine algorithm does not support vector data whose value is 0.

Write data

POST image-index/_bulk?refresh=true
{ "index": { "_id": "1" } }
{ "image-vector": [1, 5, -20], "title": "moose family", "file-type": "jpg" }
{ "index": { "_id": "2" } }
{ "image-vector": [42, 8, -15], "title": "alpine lake", "file-type": "png" }
{ "index": { "_id": "3" } }
{ "image-vector": [15, 11, 23], "title": "full moon", "file-type": "jpg" }

Perform an approximate kNN search

To perform an approximate kNN search, you must use the search API to configure kNN parameters.

Note The kNN search API is deprecated in Elasticsearch later than V8.4. You can configure the kNN parameters in the search API to perform an approximate kNN search.

POST image-index/_search
{
  "knn": {
    "field": "image-vector",
    "query_vector": [-5, 9, -12],
    "k": 10,
    "num_candidates": 100
  },
  "fields": [ "title", "file-type" ]
}

The following table describes the kNN parameters. For more information, see Search API.


Parameter	Required	Description
field	Yes	The name of the vector field that you want to search.
query_vector	Yes	The query vector. The query vector must have the same number of dimensions as the vector field specified by the field parameter.
k	Yes	The number of nearest neighbors to return. The value of the k parameter must be less than the value of the num_candidates parameter.
num_candidates	Yes	The number of nearest neighbor candidates that need to be searched per shard. The value cannot exceed 10000. Note The larger the value of the num_candidates parameter, the higher the accuracy of the value of the k parameter. However, the search speed slows down.
filter	No	The Domain Specific Language (DSL) statement used to filter documents. The approximate kNN search returns the top k documents that match the filter condition. If you do not configure this parameter, all documents are matched.

Exact kNN search

Create an index

PUT zl-index
{
  "mappings": {
    "properties": {
      "product-vector": {
        "type": "dense_vector",
        "dims": 5,
        "index": false
      },
      "price": {
        "type": "long"
      }
    }
  }
}

The following table describes some of the parameters that are used to define settings related to fields of the dense_vector type. For more information, see dense-vector.


Parameter	Description
type	The data type of fields that are used to store floating-point numbers. Set this parameter to dense_vector.
dim	The number of dimensions for each vector.
index	Specifies whether to generate a new index for the exact kNN search. Default value: false. For an exact kNN search, the index parameter is optional. You can leave this parameter empty or set this parameter to false to improve the efficiency of the exact kNN search.

Write data

POST zl-index/_bulk?refresh=true
{ "index": { "_id": "1" } }
{ "product-vector": [230.0, 300.33, -34.8988, 15.555, -200.0], "price": 1599 }
{ "index": { "_id": "2" } }
{ "product-vector": [-0.5, 100.0, -13.0, 14.8, -156.0], "price": 799 }
{ "index": { "_id": "3" } }
{ "product-vector": [0.5, 111.3, -13.0, 14.8, -156.0], "price": 1099 }

Perform an exact kNN search

The following code shows how to specify the vector function cosineSimilarity in the script_score query and configure a filter condition in the script_score.query parameter to limit the number of documents that can be passed to the vector function.

POST zl-index/_search
{
  "query": {
    "script_score": {
      "query" : {
        "bool" : {
          "filter" : {
            "range" : {
              "price" : {
                "gte": 1000
              }
            }
          }
        }
      },
      "script": {
        "source": "cosineSimilarity(params.queryVector, 'product-vector') + 1.0",
        "params": {
          "queryVector": [-0.5, 90.0, -10, 14.8, -156.0]
        }
      }
    }
  }
}

The following table describes the vector functions that are supported by the script_score query. For more information, see Functions for vector fields in the documentation for open source Elasticsearch.


Function	Description
cosineSimilarity	Calculates the cosine similarity between a query vector and a document vector.
dotProduct	Calculates the dot product of a query vector and a document vector.
l1norm	Calculates the L1 distance (Manhattan distance) between a query vector and a document vector.
l2norm	Calculates the L2 distance (Euclidean distance) between a query vector and a document vector.