Elasticsearch:Aliyun-knn vector search plugin

Prerequisites

• The installation method for the aliyun-knn plug-in varies based on the version of your Alibaba Cloud Elasticsearch cluster and its kernel version. For details, see the following table.

  Elasticsearch version	Kernel version	Description
  6.7.0	Earlier than 1.2.0	You must manually install the aliyun-knn plug-in on the Plug-ins page. For more information, see Install or uninstall default plug-ins. Features such as script-based search and index warm-up are not supported. If you need these features, use a cluster that has an Alibaba Cloud Elasticsearch kernel version. For more information, see Kernel version release notes. When you create a vector index, only the default SquaredEuclidean distance metric is supported. You cannot specify a distance metric by using the distance_method parameter.
  6.8	N/A
  7.4	N/A
  7.7	N/A
  6.7.0	1.2.0 or later	The aliyun-knn plug-in is integrated into the apack plug-in, which is installed by default. To install or uninstall the aliyun-knn plug-in, you must manage the apack plug-in. For more information, see Use the physical replication feature of the apack plug-in. To use advanced features such as script-based search, index warm-up, and extended functions, you must upgrade the kernel version to 1.3.0 or later. For instructions, see Upgrade versions. If a mapping parsing error occurs when you create a vector index, upgrade the kernel version to 1.3.0 or later and try again.
  7.10.0	1.4.0 or later	The aliyun-knn plug-in is integrated into the apack plug-in, which is installed by default. To install or uninstall the aliyun-knn plug-in, you must manage the apack plug-in. For more information, see Use the physical replication feature of the apack plug-in. If the minor kernel version is 1.4.0 or later, the apack plug-in is the latest version and requires no update. To check the plug-in version, run the GET _cat/plugins?v command.
  Other versions	N/A	Vector search is not supported.

  Note

  The kernel version is different from the apack plug-in version. To check the apack plug-in version, run the GET _cat/plugins?v command.

• Plan your index.

  Algorithm	Use cases	In-memory	Other
  hnsw	Small single-machine datasets. Low-latency requirements. High recall requirements.	Yes	The hnsw algorithm is based on the idea that "a neighbor's neighbor is likely also a neighbor." This algorithm's distance metric must satisfy the triangle inequality (the sum of two sides of a triangle is greater than the third side). For example, inner product vector spaces do not satisfy the triangle inequality and must be converted to Euclidean or spherical spaces to use the hnsw search method. After writing data, run forceMerge periodically during off-peak hours to reduce query latency.
  linear	Brute-force search. 100% recall. Latency increases with the data volume. Use for baseline comparisons.	Yes	None.

• Plan your cluster.

  Parameter	Description
  Data node specification (Required)	16-core 64 GB or higher. Note Building an index with the aliyun-knn plug-in is resource-intensive. Low-specification clusters can become bottlenecks and affect stability. Use a cluster with 16-core 64 GB or higher specifications.
  Node type	The cluster must have dedicated master nodes.
  Cluster off-heap memory	More than double the total size of vector data in the cluster. Example: An index contains one 960-dimensional float field and 400 documents. Because a float occupies 4 bytes, the memory used for vector data is calculated as follows: 960 × 400 × 4 = 1.5 MB. Therefore, the off-heap memory must exceed 3 MB (1.5 MB × 2). Note If you perform a force merge operation, both the old and new data are loaded into memory simultaneously. In this case, you must double the value calculated by using the preceding formula. For clusters with 64 GB of memory or more, the off-heap memory size is approximately equal to the total memory minus 32 GB (off-heap memory ≈ total memory - 32 GB).
  Write throughput	Building a vector index is a CPU-intensive task. For a 16-core 64 GB data node, limit the peak write throughput to 5,000 tps per node. In addition, during a vector query, all index files are loaded into system memory. Therefore, avoid large-volume writes while serving business queries. This helps prevent shard restarts caused by memory pressure.

  Note

  These values are estimates. Base your plan on your actual workload, run load tests in advance, and provision sufficient memory.

Limitations

• The aliyun-knn plug-in is not supported on version 6.7 instances with the shared elastic storage feature enabled.

• To install the aliyun-knn plug-in, the data nodes of your Alibaba Cloud Elasticsearch instance must be 16-core 64 GB or larger. If your nodes do not meet this requirement, you must upgrade their specifications. For more information, see Upgrade the configuration of a cluster.

• The aliyun-knn plug-in is incompatible with some enhanced AliES kernel features, such as the physical replication feature. If this feature is enabled, you must disable it before using the plug-in. For instructions, see Use the physical replication feature of the apack plug-in.

• You cannot use OSS snapshots or DataWorks for data migration on instances that use the aliyun-knn plug-in. We recommend using Logstash instead.

Create a vector index

1. Log on to the Kibana console of your Elasticsearch cluster.

   For instructions, see Log on to the Kibana console.

   Note

   Examples here use Elasticsearch V6.7.0. Operations may vary slightly for other versions.

2. In the left navigation menu, choose Management > Dev Tools.

3. On the Console tab, run the following command to create a vector index.

   PUT test
   {
     "settings": {
       "index.codec": "proxima",
       "index.vector.algorithm": "hnsw"
     },
     "mappings": {
       "_doc": {
         "properties": {
           "feature": {
             "type": "proxima_vector", 
             "dim": 2, 
             "vector_type": "float", 
             "distance_method": "SquaredEuclidean" 
           }
         }
       }
     }
   }

   Note

   • You can add other Elasticsearch-supported field types to the vector index.

   • If a mapping parsing error occurs when you create the vector index: "type": "mapper_parsing_exception", "reason": "Mapping definition for [feature] has unsupported parameters: [distance_method : SquaredEuclidean]", upgrade the kernel to the latest version and try again.

   Type	Parameter	Default	Description
   setting	index.codec	proxima	Specifies whether to build a proxima kNN index at the underlying layer. Valid values: proxima (Recommended): A proxima kNN index is built at the underlying layer to support vector search. null: Does not build a proxima kNN index. Instead, it only builds a forward index. In this case, fields of the proxima_vector type support only script-based search and do not support hnsw or linear search. Note For large datasets where query latency is not a primary concern, you can remove this parameter or set it to null. This allows you to perform kNN vector queries by using script-based search. Script-based search is supported if your instance is V6.7.0 with apack plug-in version 1.2.1 or later, or if your instance is V7.10.0 with apack plug-in version 1.4.0 or later.
   	index.vector.algorithm	hnsw	The vector search algorithm. Valid values: hnsw: The HNSW algorithm. linear: The linear algorithm.
   	index.vector.general.builder.offline_mode	false	Specifies whether to use the offline optimization mode to build the kNN index. Valid values: false: Disables the offline optimization mode. true: Enables the offline optimization mode. This mode reduces segment fragmentation during writes and improves write throughput. Note To enable offline optimization mode, your instance must be V6.7.0 with apack plug-in version 1.2.1 or later, or V7.10.0 with apack plug-in version 1.4.0 or later. Indexes built in offline optimization mode do not support script-based data search. We recommend enabling offline optimization mode for one-time bulk imports of a full dataset.
   mapping	type	proxima_vector	The vector field type. For example, setting the feature field to proxima_vector specifies that feature is a vector field.
   	dim	2	The vector dimension. This parameter is required. The value must be an integer from 1 to 2048.
   	vector_type	float	The data type of the vector. Valid values: float: Floating-point. short: Short integer. binary: Binary. For the binary type, the vector data must be represented as an array of unsigned 32-bit integers (uint32), and the dim value must be a multiple of 32. For example, if your business data is the 64-bit binary value 1000100100100101111000001001111101000011010010011010011010000100, the written vector is [-1994006369, 1128900228]. Note All three vector data types are supported on instances that are V6.7.0 with apack plug-in version 1.2.1 or later, or V7.10.0 with apack plug-in version 1.4.0 or later. Earlier versions only support the float type.
   	distance_method	SquaredEuclidean	The distance function. Valid values: SquaredEuclidean: Euclidean distance (not square-rooted). InnerProduct: Inner product. Cosine: Cosine similarity. Hamming: Hamming distance (for the binary type only). Note All four distance functions are supported on instances that are V6.7.0 with apack plug-in version 1.2.1 or later, or V7.10.0 with apack plug-in version 1.4.0 or later. Earlier versions only support the default SquaredEuclidean function. You cannot specify other distance functions by using the distance_method parameter. For more information about distance functions, see Distance functions. Because the Hamming function has a special implementation, standard knn queries are not supported when you use hnsw or linear indexes. Only script-based queries that are compatible with script_score are supported. You need to test the query statements in different scenarios to ensure that they are suitable for your specific business requirements.

   Note

   • You can run the GET /_cat/plugins?v command to check the apack plug-in version. If the version does not meet the requirements, you can submit a ticket to request an upgrade.

   • kNN vector search also provides advanced query parameters. For more information, see Advanced parameters.

4. Run the following command to add a document.

   POST test/_doc
   {
     "feature": [1.0, 2.0]
   }

   Note

   For all array types except binary, the array length must match dim. For the binary type, vector data must be converted to an array of unsigned 32-bit integers (uint32), and dim must be an integer multiple of 32.

Vector search

• Standard search

  Run the following command to perform a standard search.

  GET test/_search
  {
    "query": {
      "hnsw": {  
        "feature": {
          "vector": [1.5, 2.5], 
          "size": 10 
        }
      }
    }
  }

  The following table describes the common parameters.

  Parameter	Description
  hnsw	The vector search algorithm. This must match the algorithm specified when you created the index.
  vector	The query vector. The length of the array must be the same as the dim specified in the mapping when you created the index.
  size	Specifies the number of documents to return. Note The size parameter in a vector search is different from the built-in Elasticsearch size parameter. The former controls the number of documents returned by the kNN plugin, while the latter controls the total number of documents returned by the entire query. When you run a query, the system first uses the vector search size parameter to retrieve the specified number of top documents. Then, it applies the built-in Elasticsearch size parameter to the results before returning the final set of documents. We recommend that you set the vector search size parameter to the same value as the built-in Elasticsearch size parameter, which defaults to 10.

  Note

  The kNN vector search also provides advanced query parameters. For more information, see Advanced parameters.

• Script query

  Vector script queries can only be used with script_score. For example, you can use script_score to score each returned document based on the formula 1/(1+l2Squared(params.queryVector, doc['feature'])). The following command shows an example.

  GET test/_search
  {
    "query": {
      "match_all": {}
    },
    "rescore": {
      "query": {
        "rescore_query": {
          "function_score": {
            "functions": [{
              "script_score": {
                "script": {
                    "source": "1/(1+l2Squared(params.queryVector, doc['feature'])) ", 
                    "params": {
                      "queryVector": [2.0, 2.0]
                    }
                }
              }
            }]
          }
        }
      }
    }
  }

  The kNN vector script query does not support functions provided by X-Pack. Only the following functions are supported:

  Function	Description
  l2Squared(float[] queryVector, DocValues docValues)	Euclidean distance function.
  hamming(float[] queryVector, DocValues docValues)	Hamming distance function.
  cosineSimilarity(float[] queryVector, DocValues docValues) cosine(float[] queryVector, DocValues docValues)	Cosine similarity function. Note For Alibaba Cloud Elasticsearch V6.7, use the cosineSimilarity(float[] queryVector, DocValues docValues) function. For V7.10, use the cosine(float[] queryVector, DocValues docValues) function.

  Note

  • To use the script query feature, your setup must meet one of the following requirements: cluster version 6.7.0 with apack plugin V1.2.1 or later, or cluster version 7.10.0 with apack plugin V1.4.0 or later. You can run the GET /_cat/plugins?v command to check the apack plugin version. If the version does not meet the requirements, you can submit a ticket to request an upgrade.

  • Function parameters:

    • float[] queryVector: The query vector.

    • DocValues docValues: The document vector.

  • Vector script queries are not supported for indexes built with the offline optimization mode (index.vector.builder.offlineMode = true).

• Index warm-up

  A kNN index performs a full in-memory search, which can cause high query latency from a cold start when the index is first loaded. The kNN plugin's index warm-up feature prevents this by pre-loading an index into local memory before it serves search requests, significantly reducing cold start latency.

  • Warm up all vector indexes.

    POST _vector/warmup

  • Warm up a specific vector index.

    POST _vector/{indexName}/warmup

  Note

  • To use the index warm-up feature, your setup must meet one of the following requirements: cluster version 6.7.0 with apack plugin V1.2.1 or later, or cluster version 7.10.0 with apack plugin V1.4.0 or later. You can run the GET _cat/plugins?v command to check the apack plugin version. If the version does not meet the requirements, you can submit a ticket to request an upgrade.

  • If your cluster contains many large vector indexes but your application only performs vector searches on a subset of them, warm up only those specific indexes to improve in-memory search performance.

Vector scoring

Vector search uses a unified scoring formula based on a distance metric function, which directly affects search ranking.

Scoring formula:

score = 1 / (vector distance function + 1)

Distance measurement functions

Circuit breaker parameters

Advanced parameters

Example query:

GET test/_search
{
  "query": {
    "hnsw": {
      "feature": {
        "vector": [1.5, 2.5],
        "size": 10,
        "ef": 100       
      }
    }
  }
}

FAQ

• Q: How do I evaluate the recall ratio of a query?

  A: Create two indexes with identical configurations, one using the HNSW algorithm and the other using the linear search algorithm. From a client, push the same vector data to both indexes. After the indexes are refreshed, use the same query vector to retrieve document IDs from both. The recall ratio is the number of intersecting document IDs from both indexes divided by the total number of IDs from the linear search index.

• Q: What should I do if a circuitBreakingException error occurs during cluster writes?

  A: This error indicates that off-heap memory usage has exceeded the threshold specified by indices.breaker.vector.native.indexing.limit (70% by default), triggering circuit breaking on write operations. The circuit breaker usually resets automatically after background index building completes and memory is freed. We recommend adding a retry mechanism to your client.

• Q: Why does CPU usage remain high after write operations have stopped?

  A: Vector index building occurs during the refresh or flush phases. Even after write traffic stops, background tasks for building the vector index may continue to run. These tasks release compute resources after the final refresh cycle is complete.

• Q: When querying with the aliyun-knn plug-in, I receive the following error: class_cast_exception: class org.apache.lucene.index.SoftDeletesDirectoryReaderWrapper$SoftDeletesFilterCodecReader cannot be cast to class org.apache.lucene.index.SegmentReader (org.apache.lucene.index.SoftDeletesDirectoryReaderWrapper$SoftDeletesFilterCodecReader and org.apache.lucene.index.SegmentReader are in unnamed module of loader 'app'). What should I do?

  A: Disable the physical replication feature for the index. For more information, see Use the physical replication feature of the apack plug-in.

• Q: What can I do if vector search with the aliyun-knn plug-in is slow or memory-related circuit breaking occurs?

  A: The aliyun-knn plug-in performs vector search using in-memory vectors, which is a memory-intensive process. Large indexes can cause slow performance or trigger circuit breaking when loaded into memory. As a best practice, limit your index data size to half of the available machine memory. If you cannot reduce the data volume and memory remains insufficient, upgrade the cluster configuration.

• Q: In kNN scenarios, amust_not exists query fails to filter documents with an empty 'feature' field. How can I write a query to filter for this data?

  A: The storage mechanism for kNN data is unique and may be incompatible with certain DSL queries. You can use the following script to filter the data instead.

  GET jx-similar-product-v1/_search
  {
     "query": {
       "bool": {
         "must": {
           "script": {
             "script": {
               "source": "doc['feature'].empty",
               "lang": "painless"
               }
           }
         }
       }
     }
  }