Elasticsearch:Use the aliyun-knn plug-in

How it works

The plug-in adds a proxima_vector field type and two search algorithms to Elasticsearch:

• Hierarchical Navigable Small World (HNSW): graph-based approximate kNN — fast, high-recall, in-memory. Best for low-latency queries on smaller datasets.

• Linear Search: brute-force exact kNN — always 100% recall, but latency scales with data volume.

The plug-in is compatible with all versions of open source Elasticsearch. Both algorithms store indexes in memory and are compatible with standard Elasticsearch features: multiple replica shards, distributed search, real-time incremental synchronization, near-real-time (NRT) updates, and restoration.

Performance benchmark (tested on V6.7.0, two data nodes with 16 vCPUs and 64 GiB each, 20 million SIFT 128-dimensional float vectors):

p99 = time to respond to 99% of queries.

The plug-in is used in numerous production scenarios, including Pailitao, Image Search, Youku video fingerprinting, Qutoutiao video fingerprinting, Taobao commodity recommendation, customized searches, and Crossmedia searches.

Prerequisites

Before you begin, make sure that:

• The aliyun-knn plug-in is installed and compatible with your cluster version (see the version compatibility table below).

• Data node specifications are at least 16 vCPUs and 64 GiB of memory. The plug-in creates indexes during both refresh and flush operations, which are CPU- and memory-intensive. Undersized nodes cause bottlenecks or instability.

• Your cluster has independent dedicated master nodes.

• Off-heap memory is greater than twice the total size of your vector data. During force merge, both old and new data are in memory simultaneously — plan for four times the vector data size if you run force merges. Memory estimate formula: num_vectors × num_dimensions × bytes_per_element Example: a 960-dimensional float index with 400 documents uses 960 × 400 × 4 = 1,536,000 bytes (~1.5 MB). Off-heap memory must be greater than 3 MB (1.5 × 2). For clusters with 64 GiB or more of total memory, off-heap memory ≈ total memory − 32 GiB.

• Write throughput stays below 5,000 TPS per data node (16 vCPUs, 64 GiB). Vector indexing is CPU-intensive; avoid high write concurrency while queries are running.

Choose an algorithm:

Version compatibility:

The kernel version is different from the apack plug-in version. Run GET _cat/plugins?v to check the apack plug-in version.

Limitations

• Data node specifications must be at least 16 vCPUs and 64 GiB before installing the plug-in. To upgrade, see Upgrade the configuration of a cluster.

• If the physical replication feature is enabled on your cluster, disable it before using the aliyun-knn plug-in. See Use the physical replication feature of the apack plug-in.

• Data migration via Object Storage Service (OSS) snapshots or DataWorks is not supported. Use Logstash to migrate data.

Create a vector index

Log on to the Kibana console of your Elasticsearch cluster. For details, see Log on to the Kibana console.

Creating a vector index involves three steps:

1. Configure index settings — choose the codec and algorithm.

2. Define a vector field in mappings — set the field type, dimensions, and distance function.

3. Add documents to the index.

PUT test
{
  "settings": {
    "index.codec": "proxima",
    "index.vector.algorithm": "hnsw"
  },
  "mappings": {
    "_doc": {
      "properties": {
        "feature": {
          "type": "proxima_vector",
          "dim": 2,
          "vector_type": "float",
          "distance_method": "SquaredEuclidean"
        }
      }
    }
  }
}

POST test/_doc
{
  "feature": [1.0, 2.0]
}

Search for a vector

GET test/_search
{
  "query": {
    "hnsw": {
      "feature": {
        "vector": [1.5, 2.5],
        "size": 10
      }
    }
  }
}

GET test/_search
{
  "query": {
    "match_all": {}
  },
  "rescore": {
    "query": {
      "rescore_query": {
        "function_score": {
          "functions": [{
            "script_score": {
              "script": {
                "source": "1/(1+l2Squared(params.queryVector, doc['feature']))",
                "params": {
                  "queryVector": [2.0, 2.0]
                }
              }
            }
          }]
        }
      }
    }
  }
}

POST _vector/warmup

POST _vector/{indexName}/warmup

Distance measurement functions

All distance functions use the same scoring formula:

Score = 1 / (distance + 1)

By default, SquaredEuclidean (Euclidean distance without square root extraction) is used.

All four distance functions are supported on V6.7.0 clusters with apack V1.2.1+ and V7.10.0 clusters with apack V1.4.0+. All other clusters support only SquaredEuclidean. Run GET _cat/plugins?v to check your version. The Hamming function uses a special implementation: when HNSW or Linear Search is the algorithm, standard kNN searches are not supported for Hamming indexes — only script queries are supported, compatible with the script_score parameter.

Circuit breaker parameters

These cluster-level parameters protect your cluster from memory exhaustion during vector indexing. Check current values with GET _cluster/settings. Keep the default values unless you have a specific reason to change them.

Advanced parameters

GET test/_search
{
  "query": {
    "hnsw": {
      "feature": {
        "vector": [1.5, 2.5],
        "size": 10,
        "ef": 100
      }
    }
  }
}

FAQ

How do I evaluate the recall ratio of my HNSW index?

Create two indexes with identical settings — one using HNSW and one using Linear Search. Write the same vector data to both, then run the same query vector against each. Compare the returned document IDs: recall ratio = (IDs returned by both indexes) / (total IDs returned by Linear Search).

My writes returned a `circuitBreakingException` error.

Off-heap memory usage exceeded the indices.breaker.vector.native.indexing.limit threshold (default 70%), and writes were paused. In most cases, writes resume automatically once the system finishes building indexes and frees memory. Add a retry mechanism to your write script to handle this automatically.

Why is CPU usage still high after writes are paused?

The system continues building vector indexes during refresh and flush operations even after the write pause. CPU usage drops after the final refresh completes.

I see a `class_cast_exception: class org.apache.lucene.index.SoftDeletesDirectoryReaderWrapper$SoftDeletesFilterCodecReader cannot be cast to class org.apache.lucene.index.SegmentReader` error.

Disable the physical replication feature on your index. See Use the physical replication feature of the apack plug-in.

Vector searches are slow or suspended due to high memory usage.

Vector indexes are loaded into data node memory during a search. If a data node stores more vector index data than half its total memory, searches slow down or stall. Keep each data node's vector data volume below 50% of its total memory. If node memory is insufficient, upgrade the data nodes. See Upgrade the configuration of a cluster.

Are best practices and real-world examples available?

The Alibaba Cloud developer community provides business scenarios and a best practice guide for the aliyun-knn plug-in.

I cannot use `must_not exists` to filter documents where the vector field is empty.

Vector data is stored differently from regular fields and may not be compatible with standard Query DSL filters. Use the following script query instead:

GET jx-similar-product-v1/_search
{
  "query": {
    "bool": {
      "must": {
        "script": {
          "script": {
            "source": "doc['feature'].empty",
            "lang": "painless"
          }
        }
      }
    }
  }
}