Unlock the Power of Vector Search: Alibaba Cloud Elasticsearch with the aliyun-knn Plugin

Elasticsearch has been a widely used search engine that can handle a variety of complex search tasks. With the introduction of vector search, Elasticsearch's capabilities have extended even further into the realm of machine learning and artificial intelligence. Specifically on Alibaba Cloud, the aliyun-knn plugin utilizes the vector library of Proxima, a vector search engine developed by Alibaba DAMO Academy, to enhance search functionalities.

In this article, we will delve into how to harness the power of the aliyun-knn plugin within Alibaba Cloud Elasticsearch, and demonstrate its use in scenarios such as image search, video fingerprinting, and recommendation systems. For more information on Alibaba Cloud Elasticsearch, visit the official product page

Background and Use Cases

The Alibaba Cloud Elasticsearch vector search engine, empowered by the aliyun-knn plugin, is already leveraged in many production scenarios within Alibaba Group. It's built to handle complex tasks including Pailitao image search, Youku video fingerprinting, and Taobao commodity recommendations. Additionally, it supports features characteristic of distributed searches like multiple replica shards and restoration.

Installation and Prerequisites

Before we delve into the code, ensure that the aliyun-knn plugin is installed on your Elasticsearch cluster. The plugin’s availability by default is contingent on your cluster’s version and kernel version. You can verify this by running GET _cat/plugins?v in your Elasticsearch console.
For clusters running a kernel version earlier than V1.2.0, follow the instructions provided in Install and remove a built-in plug-in

For more advanced features, you may need a cluster with a kernel version V1.3.0 or later. Data migration using snapshots in OSS or DataWorks is not supported, and Logstash is recommended as an alternative.

Creating a Vector Index

Let’s create a basic vector index on an Alibaba Cloud Elasticsearch cluster using the Kibana console.

PUT /my_vector_index
{
  "settings": {
    "index.codec": "proxima",
    "index.vector.algorithm": "hnsw"
  },
  "mappings": {
    "properties": {
      "feature": {
        "type": "proxima_vector",
        "dim": 128,
        "vector_type": "float",
        "distance_method": "SquaredEuclidean"
      }
    }
  }
}

In this example, we're creating a Proxima vector index with Euclidean distance as the distance measurement metric.

index.codec: Sets to "proxima" to enable vector search.
index.vector.algorithm: We’re using "hnsw" for the Hierarchical Navigable Small World algorithm, optimized for small volumes of in-memory data and low latency.

Adding a Document to the Index

Once the index is created, you can add a document to it. Here we are adding a two-dimensional float-type vector.

POST /my_vector_index/_doc
{
  "feature": [1.0, 2.0]
}

Replace the array [1.0, 2.0] with the actual data you wish to index.

Performing a Search with the Vector Index

Here’s how you can perform a standard search with the aliyun-knn plugin:

GET /my_vector_index/_search
{
  "query": {
    "hnsw": {
      "feature": {
        "vector": [1.5, 2.5], 
        "size": 10
      }
    }
  }
}

The size parameter defines how many matching vector documents to retrieve.

Next Steps

Using Alibaba Cloud Elasticsearch with the aliyun-knn plugin offers a fantastic opportunity to boost your search capabilities with the power of AI, especially in scenarios demanding vector-based search operations.

Interested in trying it out? Alibaba Cloud Elasticsearch offers a 30 Day Free Trial to help you get started on your search journey. Ready to start your journey with Elasticsearch on Alibaba Cloud? Explore our tailored Cloud solutions and services to take the first step towards transforming your data into a visual masterpiece.

Please Click here, Embark on Your 30-Day Free Trial

Community

Unlock the Power of Vector Search: Alibaba Cloud Elasticsearch with the aliyun-knn Plugin

Background and Use Cases

Installation and Prerequisites

Creating a Vector Index

Adding a Document to the Index

Performing a Search with the Vector Index

Next Steps

Read previous post:

Read next post:

Data Geek

You may also like

Comments

Data Geek

Related Products

Alibaba Cloud Elasticsearch

OpenSearch

Image Search