Use the aliyun-knn plug-in for vector searches

Index planning

Cluster planning

Procedure

1. Log on to the Kibana console of your Elasticsearch cluster.
   For more information, see Log on to the Kibana console.
2. In the left-side navigation pane, click Dev Tools.
3. On the Console tab of the page that appears, run the following command to create an index:
   Notice The following sample code is applicable only to Alibaba Cloud Elasticsearch V6.7.0. For information about the sample code that is applicable to Alibaba Cloud Elasticsearch V7.4.0, see open source Elasticsearch documentation.

   PUT test
   {
     "settings": {
       "index.codec": "proxima",
       "index.vector.algorithm": "hnsw" 
     },
     "mappings": {
       "_doc": {
         "properties": {
           "feature": {
             "type": "proxima_vector", 
             "dim": 2 
           },
           "id": {
             "type": "keyword"
           }
         }
       }
     }
   }

   Parameter	Description
   index.vector.algorithm	The algorithm. Valid values: hnsw and linear.
   type	The field type. Set the value to proxima_vector to specify a vector-type field.
   dim	The number of vector dimensions. Valid values: 1 to 2048.

   The preceding sample code creates an index named test. The type of the index is _doc. The index contains two fields: feature and id. You can customize the names of the index and fields based on your business requirements.

4. Run the following command to add a document:

   POST test/_doc
   {
     "feature": [1.0, 2.0], 
     "id": 1
   }

   Notice The value of the feature field must be a float array. The length of the array must be the same as that specified in the dim parameter in mapping.
5. Run the following command to search for the document:

   GET test/_search
   {
     "query": {
       "hnsw": {  
         "feature": {
           "vector": [1.5, 2.5], 
           "size": 10 
         }
       }
     }
   }

   Parameter	Description
   hnsw	The value must be the same as that of the algorithm parameter specified when you create the index.
   vector	A float array. The length of the array must be the same as that specified in the dim parameter in mapping.
   size	The number of the top-ranked documents to return.

Parameters

Sample request:

GET test/_search
{
  "query": {
    "hnsw": {
      "feature": {
        "vector": [1.5, 2.5],
        "size": 10,
        "ef": 100       
      }
    }
  }
}

FAQ

• Q: How do I evaluate the recall ratio of documents?

  A: You can create two indexes. One uses the HNSW algorithm and the other uses the Linear Search algorithm. Keep other index settings consistent for the two indexes. Use a client to add the same vector data to the indexes. Then, refresh the indexes. Compare the document IDs returned by the HNSW index and the Linear Search index after the same query vector is used. Then, find out the same document IDs that are returned by both indexes.

  Note Divide the number of document IDs returned by both indexes by the total number of returned document IDs to calculate the recall ratio of the documents.

• Q: When I write data to Elasticsearch, the system displays the circuitBreakingException error. What do I do?

  A: This error indicates that the off-heap memory usage exceeds the proportion specified by the indices.breaker.vector.native.indexing.limit parameter and that the write operation is suspended. The default proportion is 70%. In most cases, after Elasticsearch creates indexes and releases memory, the write operation is automatically resumed. We recommend that you add a retry mechanism to the data write script on your client.

• Q: Why is the CPU still working after the write operation is suspended?

  A: Elasticsearch creates vector indexes during both the refresh and flush processes. The vector index creation task may be still running even if the write operation is suspended. Computing resources are released after the final refresh is completed.