Tablestore:KNN vector query

Scenarios

The KNN vector query feature is suitable for the following scenarios:

• RAG

  RAG is an AI framework that combines retrieval capabilities with the capabilities of large language models (LLMs) to improve the accuracy of outputs generated by LLMs, especially in the field of private data or professional data. RAG is widely used in knowledge bases.

• Recommendation system

  KNN vector query can be used by platforms such as e-commerce, social media, and video streaming platforms. For example, a platform can encode content, such as user behaviors, preferences, and content features, as vectors and store the vectors. Then, KNN vector query can be used to quickly find products, articles, or videos that match user interests. This way, the platform can provide custom recommendations to improve user experience and loyalty.

• Similarity detection

  In content recognition scenarios, such as image recognition, video recognition, speech recognition, voiceprint recognition, and facial recognition, unstructured data is converted to vectors and stored. Then, the system uses KNN vector query to quickly find the most similar content. For example, after a user uploads an image of a commodity to an e-commerce platform, the system can quickly find the images of commodities that are similar to the uploaded image in style, color, or pattern.

• NLP and semantic search

  In the NLP field, text is converted to vectors by using techniques such as Word2Vec and Bidirectional Encoder Representations from Transformers (BERT). Then, KNN vector query is used to understand the semantics of query statements and find the most semantically relevant content such as documents, news, and Q&A pairs. This helps improve the relevance of query results and user experience.

• Knowledge graph and intelligent Q&A

  The nodes and relationship of nodes in knowledge graphs can be represented by vectors. Then, KNN vector query can be used to accelerate entity linking, relational inference, and the response of the intelligent Q&A system. This helps the system accurately understand the questions and provide answers to complex questions.

Benefits

Features

You can use the KNN vector query feature to perform an approximate nearest neighbor search based on vectors. This way, you can find data items that have the highest similarity to the vector that you want to query in a large-scale dataset.

KNN vector query is an out-of-the-box feature that inherits all features of search indexes. You can use the KNN vector query feature in pay-as-you-go mode, without the need to build or deploy a system. KNN vector query supports the streaming mode of search indexes. Data can be queried in near real time after the data is written to tables. KNN vector query also supports high-throughput insert, update, and delete operations. The query performance is comparable to that of systems that use the HNSW algorithm.

When you use the KNN vector query feature, you must specify the vector for which you want to query the similarity, the Vector field that you want to query, and the top K query results that have the highest similarity to the vector that you want to query. You can also use the features of other query methods in combination with KNN vector query to filter the query results.

Usage notes

When you use the KNN vector query feature, take note of the following items:

• Limits are imposed on the number of Vector fields and the number of dimensions for a Vector field. For more information, see Search index limits.

• The search index server has multiple partitions. Each partition of the search index server returns the top K neighbors nearest to the vector that you want to query. The top K nearest neighbors returned by the partitions are aggregated on the client node. If you want to use tokens to query all data by page, the total number of rows in the response is related to the number of partitions of the search index server.

• The KNN vector query feature is supported in the following regions: China (Hangzhou), China (Shanghai), China (Qingdao), China (Beijing), China (Zhangjiakou), China (Ulanqab), China (Shenzhen), China (Guangzhou), China (Chengdu), China (Hong Kong), Japan (Tokyo), Singapore, Malaysia (Kuala Lumpur), Indonesia (Jakarta)), Philippines (Manila), Thailand (Bangkok), Germany (Frankfurt), UK (London), US (Virginia), US (Silicon Valley), and SAU (Riyadh - Partner Region).

Procedure

1. Generate vectors and write data to Tablestore. For more information, see Generate vectors.

   Use open source models to convert Tablestore data into vectors and store the vectors.

2. Create a search index and specify a Vector field. For more information, see Create a search index.

   When you create a search index, specify the data type of the Vector field, the number of dimensions of the Vector field, and the algorithm that you want to use to measure the distance between vectors.

3. Use the KNN vector query feature to query data. For more information, see Use KNN vector query.

Billing rules

During the public preview, you are not charged for the billable items specific to the KNN vector query feature when you use the feature. You are charged for other billable items based on the existing billing rules.

When you use a search index to query data, you are charged for the read throughput that is consumed. For more information, see Billable items of search indexes.

Appendix 1: Combination of KNN vector query and Boolean query

You can use KNN vector query and Boolean query in various combinations. The query performance varies based on the combination that you use. In the example in this section, a small amount of data meets the filter conditions.

In this example, 0.1 billion images are stored in a table and 50,000 images belong to User A. Among the 50,000 images, 50 images are stored in the previous seven days. User A wants to search for 10 images that have the highest similarity to the specified image among the 50 images. The following table describes two common combinations of KNN vector query and Boolean query that User A can use to meet the query requirements.

Appendix 2: Sample vector normalization

The following sample code provides an example on how to normalize vectors:

  public static float[] l2normalize(float[] v, boolean throwOnZero) {
    double squareSum = 0.0f;
    int dim = v.length;
    for (float x : v) {
      squareSum += x * x;
    }
    if (squareSum == 0) {
      if (throwOnZero) {
        throw new IllegalArgumentException("can't normalize a zero-length vector");
      } else {
        return v;
      }
    }
    double length = Math.sqrt(squareSum);
    for (int i = 0; i < dim; i++) {
      v[i] /= length;
    }
    return v;
  }