OpenSearch:Hybrid query based on vectors and text

Query syntax

URL

curl -X POST /search or curl -X POST /vector-service/search

• The sample URL omits information such as parameters in request headers and the encoding method.

• The sample URL also omits the endpoint that is used to connect to an OpenSearch instance.

• For more information about the definitions, usage, and example values of all the request parameters that are concatenated in the preceding URL, see the "Request parameters" section of this topic.

Protocol

HTTP

Request method

POST

Supported format

JSON

Signature method

You can use the following method to calculate the request signature. The request signature is stored in the authorization header.

import com.aliyun.darabonba.encode.Encoder;
import com.aliyun.darabonbastring.Client;

public class GenerateAuthorization {

    public static void main(String[] args) throws Exception {
        String accessUserName = "username";
        String accessPassWord = "password";
        String realmStr = "" + accessUserName + ":" + accessPassWord + "";
        String authorization = Encoder.base64EncodeToString(Client.toBytes(realmStr, "UTF-8"));
        System.out.println(authorization);
    }
}

Valid format for the value of the authorization header:

cm9vdDp******mdhbA==

You must add the Basic prefix when you specify the authorization header in an HTTP request.

Example:

authorization: Basic cm9vdDp******mdhbA==

Request parameters

Usage notes

• kNN only supports single-vector queries, and the query parameters are the same as those of API queries.

• The vector scores must be converted.

  To combine the kNN and text scores, you must convert the kNN vector scores.

  # Euclidean distance
  score = 1.0 / (1.0 + l2_distance^2)
  
  # Inner product distance
  score = (1.0 + ip_distance) / 2.0

• The order parameter in the kNN query does not take effect. The order parameter at the outer layer is used.

• Default sorting order

  • Configuration method: Do not configure the rank parameter, or leave the rank parameter empty.

    {
      "rank": {}
    }

  • Combine the scores of documents with the same primary key in the two result sets based on weight. Sort the results based on the weighted scores.

    score(i) = knn_score(i) * knn_weight + text_score(i) * text_weight

• RRF

  • Configuration method:

    {
      "rank": {
        "rrf": {
          "rankConstant": 60
        }
      }
    }
    
    # rankConstant is optional. Default value: 60.

  • Formula

    score = 0.0
    if d in result(q):
        score += 1.0 / (rankConstant + rank(result(q), d))
    return score
    
    # rankConstant is a sorting constant that determines the extent to which the ranking of the documents in a result set influences the final score. The larger the value, the greater the impact of the low-ranked documents on the final score. The value of this parameter must be an integer greater than or equal to 1. Default value: 60.
    # rank(result(q), d) indicates the position of the document in a result set, starting from 1.

Query example

{
    "tableName": "test",
    "text": {
        "queryString": "title:'Alibaba'",
        "queryParams": {
            "default_op": "OR"
        },
        "filter": "count > 0",
        "terminateAfter": 100000
    },
    "knn": {
        "vector": [0.1, 0.2, 0.3, 0.4, 0.5],
        "namespace": "1",
        "topK": 100
    },
    "order": "DESC",
    "size": 10,
    "rank": {
        "rrf": {
            "rankConstant": 1
        }
    },
    "outputFields": ["title"]
}

Returned results

{
  "totalTime": 8.522,
  "coveredPercent": 1.0,
  "totalCount": 5,
  "result": [
    {
      "__source__": 2,
      "score": 0.833333,
      "namespace": 1,
      "id": 2,
      "fields": {
        "title": "a b c"
      }
    },
    {
      "__source__": 3,
      "score": 0.666666,
      "namespace": 1,
      "id": 1,
      "fields": {
        "title": "a b"
      }
    },
    {
      "__source__": 3,
      "score": 0.333333,
      "namespace": 2,
      "id": 5,
      "fields": {
        "title": "c"
      }
    },
    {
      "__source__": 2,
      "score": 0.25,
      "namespace": 2,
      "id": 4,
      "fields": {
        "title": "b c"
      }
    },
    {
      "__source__": 2,
      "score": 0.2,
      "namespace": 1,
      "id": 0,
      "fields": {
        "title": "a"
      }
    }
  ]
}

Result parameters: