Use the aliyun-knn plug-in

Index planning

Cluster planning

Create a vector index

1. Log on to the Kibana console of your Alibaba Cloud Elasticsearch cluster.
   For more information, see Log on to the Kibana console.
2. In the left-side navigation pane, click Dev Tools.
3. On the Console tab of the page that appears, run the following command to create a vector index:

   PUT test
   {
     "settings": {
       "index.codec": "proxima",
       "index.vector.algorithm": "hnsw"
     },
     "mappings": {
       "_doc": {
         "properties": {
           "feature": {
             "type": "proxima_vector", 
             "dim": 2, 
             "vector_type": "float", 
             "distance_method": "SquaredEuclidean" 
           }
         }
       }
     }
   }

   Category	Parameter	Default value	Description
   setting	index.codec	proxima	Specifies whether to create the proxima vector index. Valid values: proxima: The system creates the proxima vector index. This index supports vector searches. We recommend that you set this parameter to proxima. null: The system does not create the proxima vector index but creates only forward indexes. In this case, fields of the proxima_vector type do not support HNSW- or Linear Search-based queries. These fields support only script queries. Note If your cluster stores large volumes of data and you do not have high requirements on the latency of queries, you can remove the parameter or set the parameter to null. In this case, you can use the script query feature to search for the k-nearest neighbors (k-NN) of a vector. This feature is available only for V6.7.0 clusters whose apack plug-in is of V1.2.1 or later and V7.10.0 clusters whose apack plug-in is of V1.4.0 or later.
   	index.vector.algorithm	hnsw	The algorithm that is used to search for vectors. Valid values: hnsw: HNSW linear: Linear Search
   	index.vector.general.builder.offline_mode	false	Specifies whether to use the offline optimization mode to create the vector index. Valid values: false: The offline optimization mode is not used. true: The offline optimization mode is used. In this case, the number of segments that are written to the vector index is significantly reduced. This improves write throughput. Note The offline optimization mode is available only for V6.7.0 clusters whose apack plug-in is of V1.2.1 or later and V7.10.0 clusters whose apack plug-in is of V1.4.0 or later. You are not allowed to perform script queries on the indexes for which the offline optimization mode is used. If you want to write all of your data to your cluster at a time, we recommend that you use the offline optimization mode.
   mapping	type	proxima_vector	The field type. For example, if you set the type parameter to proxima_vector for the feature field, the feature field is a vector-type field.
   	dim	2	The number of vector dimensions. This parameter is required. Valid values: 1 to 2048.
   	vector_type	float	The data type of vectors. Valid values: float short binary If you set this parameter to binary, the vector data that you want to write to the vector index must be of the uint32 data type. This indicates that the vector data must be represented by an unsigned 32-bit decimal array. In addition, the value of the dim parameter must be a multiple of 32. For example, if you want to write the 64-bit binary business data 1000100100100101111000001001111101000011010010011010011010000100 to your vector index, set the vector parameter to [-1994006369, 1128900228] when you run the related command to add a document. Notice Only V6.7.0 clusters whose apack plug-in is of V1.2.1 or later and V7.10.0 clusters whose apack plug-in is of V1.4.0 or later support all the preceding three data types. Other clusters support only the float data type.
   	distance_method	SquaredEuclidean	The function that is used to calculate the distance between vectors. Valid values: SquaredEuclidean: calculates the Euclidean distance without square root extraction. InnerProduct: calculates the inner product. Cosine: calculates the cosine similarity. Hamming: calculates the Hamming distance. This function is available only when you set the vector_type parameter to binary. Notice Only V6.7.0 clusters whose apack plug-in is of V1.2.1 or later and V7.10.0 clusters whose apack plug-in is of V1.4.0 or later support all the preceding four functions. Other clusters support only the SquaredEuclidean function. For more information about the preceding functions, see Distance measurement functions.

   Notice

   • You can run the GET _cat/plugins?v command to obtain the version of the apack plug-in. If the version of the apack plug-in does not meet the requirements, you can submit a ticket to ask Alibaba Cloud engineers to update your apack plug-in.
   • The aliyun-knn plug-in also supports some advanced search parameters. For more information, see Advanced parameters.
4. Run the following command to add a document:

   POST test/_doc
   {
     "feature": [1.0, 2.0], 
     "id": 1
   }

   Notice If you set the vector_type parameter to binary when you create the vector index, the vector data that you want to write to the vector index must be of the uint32 data type. This indicates that you must convert your data to an unsigned 32-bit decimal array before you add a document. In addition, the value of the dim parameter must be a multiple of 32. If you set the vector_type parameter to a value other than binary, the length of your vector data must be the same as the value specified for the dim parameter.

Search for a vector

• Standard search

  Run the following command to perform a standard search for a vector:

  GET test/_search
  {
    "query": {
      "hnsw": {  
        "feature": {
          "vector": [1.5, 2.5], 
          "size": 10 
        }
      }
    }
  }

  The following table describes the parameters in the preceding command.

  Parameter	Description
  hnsw	The algorithm that is used to search for the vector. The value must be the same as that of the algorithm parameter specified when you create the index.
  vector	The vector for which you want to search. The length of the array for the vector must be the same as the value of the dim parameter specified in mapping.
  size	The number of recalled documents. Note The size parameter in a vector search command is different from the size parameter provided by Elasticsearch. The former controls the number of documents recalled by the aliyun-knn plug-in, whereas the latter controls the number of documents recalled for a search. When you perform a vector search, the system recalls the top N documents based on the value of the size parameter in the related vector search command and recalls all the matching documents based on the value of the size parameter provided by Elasticsearch. Then, the system returns results based on the top N documents and all the matching documents. We recommend that you set the two size parameters to the same value. The default value of the size parameter provided by Elasticsearch is 10.

  Note The aliyun-knn plug-in also supports some advanced search parameters. For more information, see Advanced parameters.
• Script query

  Script queries are compatible with Elastic domain-specific language (DSL) queries. For example, you can use the script_score parameter to score each document in a query response. The score is calculated by using the following formula: 1/(1 + l2Squared(params.queryVector, doc['feature'])). The following code provides a sample script query:

  GET test/_search
  {
    "query": {
      "match_all": null
    },
    "rescore": {
      "query": {
        "rescore_query": {
          "function_score": {
            "functions": [{
              "script_score": {
                "script": {
                    "source": "1/(1+l2Squared(params.queryVector, doc['feature'])) ", 
                    "params": {
                      "queryVector": [2.0, 2.0]
                    }
                }
              }
            }]
          }
        }
      }
    }
  }

  In addition to the functions supported by script queries in open source Elasticsearch, the script queries in Alibaba Cloud Elasticsearch support the functions described in the following table.

  Function	Description
  l2Squared(float[] queryVector, DocValues docValues)	The function that is used to search for a vector based on a Euclidean distance.
  hamming(float[] queryVector, DocValues docValues)	The function that is used to search for a vector based on a Hamming distance.
  cosineSimilarity(float[] queryVector, DocValues docValues) cosine(float[] queryVector, DocValues docValues)	The function that is used to search for a vector based on a cosine similarity. Note We recommend that you use the cosineSimilarity(float[] queryVector, DocValues docValues) function for V6.7 clusters and use the cosine(float[] queryVector, DocValues docValues) function for V7.10 clusters.

  Note

  • You can perform script queries only in V6.7.0 clusters whose apack plug-in is of V1.2.1 or later and V7.10.0 clusters whose apack plug-in is of V1.4.0 or later. You can run the GET _cat/plugins?v command to obtain the version of the apack plug-in. If the version of the apack plug-in does not meet the requirements, you can submit a ticket to ask Alibaba Cloud engineers to update your apack plug-in.
  • Parameters in the preceding functions:
    • float[] queryVector: the query vector. You can set this parameter to a formal or actual parameter.
    • DocValues docValues: the document vectors.
  • You are not allowed to perform script queries on the indexes that are created by using the offline optimization mode.
• Index warm-up

  Searches are performed on vector indexes in in-memory storage. Therefore, after you write data to a vector index, the latency of the first search is high. This is because the index is still being loaded to memory. To address this issue, the aliyun-knn plug-in provides the index warm-up feature. This feature allows you to warm up your vector indexes and load them to the memory of your on-premises machine before the aliyun-knn plug-in provides vector search services. This significantly reduces the latency of vector searches.

  • Run the following command to warm up all vector indexes:

    POST _vector/warmup

  • Run the following command to warm up a specific vector index:

    POST _vector/{indexName}/warmup

  Notice

  • The index warm-up feature is available only for V6.7.0 clusters whose apack plug-in is of V1.2.1 or later and V7.10.0 clusters whose apack plug-in is of V1.4.0 or later. You can run the GET _cat/plugins?v command to obtain the version of the apack plug-in. If the version of the apack plug-in does not meet the requirements, you can submit a ticket to ask Alibaba Cloud engineers to update your apack plug-in.
  • If your cluster stores a large number of vector indexes, the indexes store large volumes of data, and vector searches are required only for a specific vector index, we recommend that you warm up only the specific vector index to improve the search performance.

Vector scoring

A unified scoring formula is provided for vector searches. The formula depends on distance measurement functions. The distance measurement functions affect the sorting of search results.

Scoring formula:

Score = 1/(Vector distance + 1)

Distance measurement functions

Circuit breaker parameters

Advanced parameters

The following code provides a sample search request:

GET test/_search
{
  "query": {
    "hnsw": {
      "feature": {
        "vector": [1.5, 2.5],
        "size": 10,
        "ef": 100       
      }
    }
  }
}

FAQ

• Q: How do I evaluate the recall ratio of documents?

  A: You can create two indexes. One uses the HNSW algorithm and the other uses the Linear Search algorithm. Keep the other index settings consistent for the two indexes. Use a client to add the same vector data to the indexes, and refresh the indexes. Compare the document IDs returned by the HNSW index and the Linear Search index after the same query vector is used. Then, find out the same document IDs that are returned by both indexes.

  Note Divide the number of document IDs that both indexes return by the total number of returned document IDs to calculate the recall ratio of the documents.

• Q: When I write data to my cluster, the system returns the "circuitBreakingException" error. What do I do?

  A: This error indicates that the off-heap memory usage exceeds the proportion specified by the indices.breaker.vector.native.indexing.limit parameter and that the write operation is suspended. The default proportion is 70%. In most cases, the write operation is automatically resumed after the system creates indexes and releases memory. We recommend that you add a retry mechanism to the data write script on your client.

• Q: Why is the CPU still working after the write operation is suspended?

  A: The system creates vector indexes during both the refresh and flush processes. The index creation task may be still running even if the write operation is suspended. Computing resources are released after the final refresh is complete.

• Q: Is the best practice for the aliyun-knn plug-in provided?

  A: The Alibaba Cloud developer community provides the business scenarios and best practice of the aliyun-knn plug-in.