Use the Java Low Level REST Client to access the vector engine - Lindorm

The Java Low Level REST Client is the basic REST client provided by Elasticsearch. The API operations of the client do not encode or decode data. The Lindorm vector engine is compatible with the Elasticsearch protocol and supports a combination of scalar, vector, and full-text search capabilities. If you want to customize how to process requests and responses, you can use the Java Low Level REST Client to access the vector engine.

Prerequisites

Java Development Kit (JDK) 1.8 or later is installed.
The vector engine is activated. For more information about how to activate the vector engine, see Activate the vector engine.
LindormSearch is activated. For more information, see Activate LindormSearch.
The IP address of your client is added to the whitelist of the Lindorm instance. For more information, see Configure whitelists.

Preparations

Install the Java Low Level REST Client

For example, you can add the following dependencies to the pom.xml file in your Maven project. Sample code:

<dependency>
  <groupId>org.elasticsearch.client</groupId>
  <artifactId>elasticsearch-rest-client</artifactId>
  <version>7.10.0</version>
</dependency>
<dependency>
  <groupId>org.apache.logging.log4j</groupId>
  <artifactId>log4j-core</artifactId>
  <version>2.8.2</version>
</dependency>
<dependency>
  <groupId>org.apache.logging.log4j</groupId>
  <artifactId>log4j-api</artifactId>
  <version>2.7</version>
</dependency>

Connect to LindormSearch

// Specify the LindormSearch endpoint for Elasticsearch.
String search_url = "ld-t4n5668xk31ui****-proxy-search-public.lindorm.rds.aliyuncs.com";
int search_port = 30070;

// Specify the username and password.
String username = "user";
String password = "test";
final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(username, password));
RestClientBuilder restClientBuilder = RestClient.builder(new HttpHost(search_url, search_port));
restClientBuilder.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
  @Override
  public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
    return httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
  }
});

Parameters

Parameter	Description
search_url	The LindormSearch endpoint for Elasticsearch. For more information about how to obtain the endpoint, see View endpoints. Important If your application is deployed on an ECS instance, we recommend that you use a VPC to connect to the Lindorm instance to ensure higher security and lower network latency. If your application is deployed on a local server and needs to connect to the Lindorm instance over the Internet, you can perform the following steps to enable the public endpoint of the instance in the Lindorm console: In the left-side navigation pane, click Database Connections. On the page that appears, click the Search Engine tab. Then, click Enable Public Endpoint in the upper-right corner. If you use a VPC to access the Lindorm instance, specify the LindormSearch VPC endpoint for Elasticsearch in the value of search_url. If you use the Internet to access the Lindorm instance, specify the LindormSearch Internet endpoint for Elasticsearch in the value of search_url.
search_port	The port used to access the LindormSearch endpoint for Elasticsearch. The value of this parameter is fixed to 30070.
username	The username and password used to access LindormSearch. You can perform the following steps to obtain the default username and password: In the left-side navigation pane, click Database Connections. On the page that appears, click the Search Engine tab. Then, view the username and password displayed on this tab.
password

Create a vector index

HNSW index

The following sample code provides an example on how to create the vector_test index:

// Create an index.
Request indexRequest = new Request("PUT", "/" + indexName);
indexRequest.setJsonEntity("{\n" +
  " \"settings\" : {\n" +
  "    \"index\": {\n" +
  "      \"number_of_shards\": 2,\n" +
  "      \"knn\": true\n" +
  "    }\n" +
  "  },\n" +
  "  \"mappings\": {\n" +
  "    \"_source\": {\n" +
  "      \"excludes\": [\"vector1\"]\n" +
  "    },\n" +
  "    \"properties\": {\n" +
  "      \"vector1\": {\n" +
  "        \"type\": \"knn_vector\",\n" +
  "        \"dimension\": 3,\n" +
  "        \"method\": {\n" +
  "          \"engine\": \"lvector\",\n" +
  "          \"name\": \"hnsw\", \n" +
  "          \"space_type\": \"l2\",\n" +
  "          \"parameters\": {\n" +
  "            \"m\": 24,\n" +
  "            \"ef_construction\": 500\n" +
  "         }\n" +
  "       }\n" +
  "      },\n" +
  "      \"field1\": {\n" +
  "        \"type\": \"long\"\n" +
  "      }\n" +
  "    }\n" +
  "  }\n" +
  "}");
Response response = restClient.performRequest(indexRequest);
String responseBody = EntityUtils.toString(response.getEntity());

IVFPQ index

The following sample code provides an example on how to create the vector_ivfpq_test index:

String indexName = "vector_ivfpq_test";
Request indexRequest = new Request("PUT", "/" + indexName);
int dim = 3;
String createIndexJson = "{\n" +
  "  \"settings\": {\n" +
  "    \"index\": {\n" +
  "      \"number_of_shards\": 4,\n" +
  "      \"knn\": true,\n" +
  "      \"knn.offline.construction\": true\n" +
  "    }\n" +
  "  },\n" +
  "  \"mappings\": {\n" +
  "    \"_source\": {\n" +
  "      \"excludes\": [\"vector1\"]\n" +
  "    },\n" +
  "    \"properties\": {\n" +
  "      \"vector1\": {\n" +
  "        \"type\": \"knn_vector\",\n" +
  "        \"dimension\": %d,\n" +
  "        \"data_type\": \"float\",\n" +
  "        \"method\": {\n" +
  "          \"engine\": \"lvector\",\n" +
  "          \"name\": \"ivfpq\",\n" +
  "          \"space_type\": \"cosinesimil\",\n" +
  "          \"parameters\": {\n" +
  "            \"m\": %d,\n" +
  "            \"nlist\": 10000,\n" +
  "            \"centroids_use_hnsw\": true,\n" +
  "            \"centroids_hnsw_m\": 48,\n" +
  "            \"centroids_hnsw_ef_construct\": 500,\n" +
  "            \"centroids_hnsw_ef_search\": 200\n" +
  "          }\n" +
  "        }\n" +
  "      },\n" +
  "      \"field1\": {\n" +
  "        \"type\": \"long\"\n" +
  "      }\n" +
  "    }\n" +
  "  }\n" +
  "}"

createIndexJson = String.format(createIndexJson, dim, dim);
indexRequest.setJsonEntity(createIndexJson);
Response response = restClient.performRequest(indexRequest);
String responseBody = EntityUtils.toString(response.getEntity());
System.out.println("responseBody = " + responseBody);

Sparse vector index

The following sample code provides an example on how to create the vector_sparse_test index:

String indexName = "vector_sparse_test";

// Create an index.
Request indexRequest = new Request("PUT", "/" + indexName);
indexRequest.setJsonEntity("{\n" +
  " \"settings\" : {\n" +
  "    \"index\": {\n" +
  "      \"number_of_shards\": 2,\n" +
  "      \"knn\": true\n" +
  "    }\n" +
  "  },\n" +
  "  \"mappings\": {\n" +
  "    \"_source\": {\n" +
  "      \"excludes\": [\"vector1\"]\n" +
  "    },\n" +
  "    \"properties\": {\n" +
  "      \"vector1\": {\n" +
  "        \"type\": \"knn_vector\",\n" +
  "        \"data_type\": \"sparse_vector\",\n" +
  "        \"method\": {\n" +
  "          \"engine\": \"lvector\",\n" +
  "          \"name\": \"sparse_hnsw\",\n" +
  "          \"space_type\": \"innerproduct\",\n" +
  "          \"parameters\": {\n" +
  "            \"m\": 24,\n" +
  "            \"ef_construction\": 200\n" +
  "         }\n" +
  "       }\n" +
  "      },\n" +
  "      \"field1\": {\n" +
  "        \"type\": \"long\"\n" +
  "      }\n" +
  "    }\n" +
  "  }\n" +
  "}");
Response response = restClient.performRequest(indexRequest);
String responseBody = EntityUtils.toString(response.getEntity());
System.out.println("responseBody = " + responseBody);

Write data

You can write data to indexes that contain vector columns in the same manner as you write data to common indexes.

Write a single row of data

The following sample code provides an example on how to write data to the vector_test index:

String indexName = "vector_test";
String documentId = "1";
String jsonString = "{ \"field1\": 1, \"vector1\": [1.2, 1.3, 1.4] }";
Request request = new Request(
  "PUT", // If the document ID is specified, use the PUT method.
  "/" + indexName + "/_doc/" + documentId);
request.setJsonEntity(jsonString);
response = restClient.performRequest(request);
responseBody = EntityUtils.toString(response.getEntity());
System.out.println("writeDoc responseBody = " + responseBody);

Write multiple rows of data at the same time

// Write multiple rows of data at the same time.
Random random = new Random();
Request bulkRequest = new Request("POST", "/_bulk");
StringBuilder bulkJsonBuilder = new StringBuilder();
for (int i = 2; i < 10; i++) {
  // Replace the fields and values in the sample code with actual ones in your business.
  bulkJsonBuilder.append("{\"index\":{\"_index\":\"").append(indexName).append("\",\"_id\":\"").append(i).append("\"}}").append("\n");
  String value = String.valueOf(random.nextInt());
  float[] floatArray = {random.nextFloat(), random.nextFloat(), random.nextFloat()};
  String floatArrayString = Arrays.toString(floatArray);
  System.out.println(i + " " + value + " " + floatArrayString);
  bulkJsonBuilder.append("{\"field1\":\"").append(value).append("\",\"vector1\":\"").append(floatArrayString).append("\"}").append("\n");
}
bulkRequest.setJsonEntity(bulkJsonBuilder.toString());
response = restClient.performRequest(bulkRequest);
responseBody = EntityUtils.toString(response.getEntity());
System.out.println("bulkWriteDoc responseBody = " + responseBody);

// Initiate a refresh request to display the written data.
response = restClient.performRequest(new Request("POST", "/" + indexName + "/_refresh"));
responseBody = EntityUtils.toString(response.getEntity());
System.out.println("responseBody = " + responseBody);

Write a sparse vector

You can use the preceding method to write a sparse vector. In this case, you must modify the format of vector1.

// Write a single data entry.
String documentId = "1";
String jsonString = "{ \"field1\": 1, \"vector1\": {\"indices\": [10, 12, 16], \"values\": [1.2, 1.3, 1.4]} }";
Request request = new Request(
  "PUT", // If the document ID is specified, use the PUT method.
  "/" + indexName + "/_doc/" + documentId);
request.setJsonEntity(jsonString);
response = restClient.performRequest(request);
responseBody = EntityUtils.toString(response.getEntity());
System.out.println("writeDoc responseBody = " + responseBody);

Build an index

Important

By default, the index.knn.offline.construction parameter is set to false for all index types other than IVFPQ indexes. This way, online indexing is implemented without manual intervention.
When you create an IVFPQ index, you must explicitly set the index.knn.offline.construction parameter to true. When you trigger the index building, make sure that a sufficient amount of data is written. The data volume must exceed 256 records and be more than 30 times the value of the nlist parameter.
After you manually build the index, data can be written and queried. You do not need to build the index again.

Trigger the index building

The following sample code provides an example on how to build the vector_ivfpq_test index:

// Build an index.
Request buildIndexRequest = new Request("POST", "/_plugins/_vector/index/build");
String jsonString = "{ \"indexName\": \"vector_ivfpq_test\", \"fieldName\": \"vector1\", \"removeOldIndex\": \"true\" }";
response = restClient.performRequest(buildIndexRequest);
responseBody = EntityUtils.toString(response.getEntity());
System.out.println("buildIndex responseBody = " + responseBody);

Parameters

Parameter	Required	Description
indexName	Yes	The table name. Example: `vector_ivfpq_test`.
fieldName	Yes	The field for which you want to build the index. Example: `vector1`.
removeOldIndex	Yes	Specifies whether to delete the existing index when you build an index. Valid values: true: deletes the existing index data when the index building is triggered. You can perform the kNN query only after the building is complete. Important In the actual business, we recommend that you set the value to `true`. false (default value): retains the existing index. This may affect the search performance.

Sample output:

{
  "payload": ["default_vector_ivfpq_test_vector1"]
}

The output is the task ID generated when the index is built.

View the index status

// View the index status.
Request buildIndexRequest = new Request("GET", "/_plugins/_vector/index/tasks");
String jsonString = "{ \"indexName\": \"vector_ivfpq_test\", \"fieldName\": \"vector1\", \"taskIds\": \"[default_vector_ivfpq_test_vector1]\" }";
buildIndexRequest.setJsonEntity(jsonString);
Response response = restClient.performRequest(buildIndexRequest);
String responseBody = EntityUtils.toString(response.getEntity());
System.out.println("queryBuildIndex responseBody = " + responseBody);

taskIds indicates a list of task IDs generated when the index building process is triggered. You can provide an empty array for taskIds, such as \"taskIds\": \"[]\". This has the same effect as providing the specific task IDs.

Sample output:

{
  "payload": ["task: default_vector_ivfpq_test_vector1, stage: FINISH, innerTasks: xxx, info: finish building"]
}

The stage field indicates the index building status. Valid values: START, TRAIN, BUILDING, ABORT, FINISH, and FAIL.

Note

In most cases, you can call the /index/abort operation to stop building indexes. After the operation is complete, the value of the stage field is ABORT in the result.

Stop the index building

Stops the index building process. You cannot stop building indexes in the FINISH state.

// Stop creating an index.
Request buildIndexRequest = new Request("POST", "/_plugins/_vector/index/tasks/abort");
String jsonString = "{ \"indexName\": \"vector_ivfpq_test\", \"fieldName\": \"vector1\", \"taskIds\": \"[\"default_vector_ivfpq_test_vector1\"]\" }";
buildIndexRequest.setJsonEntity(jsonString);
Response response = restClient.performRequest(buildIndexRequest);
String responseBody = EntityUtils.toString(response.getEntity());
System.out.println("abortBuildIndex responseBody = " + responseBody);

Sample output:

{
  "payload":["Task: default_vector_ivfpq_test_vector1 remove success"]
}

Query data

Query pure vector data

You can use the k-nearest neighbors (kNN) structure to query pure vector data.

// Perform the kNN query.
Request searchRequest = new Request("GET", "/" + indexName + "/_search");
jsonString = "{"
  + "\"size\": 10,"
  + "\"query\": {"
  +     "\"knn\": {"
  +         "\"vector1\": {"
  +             "\"vector\": [2.2, 2.3, 2.4],"
  +             "\"k\": 10"
  +         "}"
  +     "}"
  + "},"
  + "\"ext\": {\"lvector\": {\"min_score\": \"0.1\"}}"
  + "}";
searchRequest.setJsonEntity(jsonString);
response = restClient.performRequest(searchRequest);
responseBody = EntityUtils.toString(response.getEntity());
System.out.println("search responseBody = " + responseBody);

Parameters

Parameter structure	Parameter	Required	Description
knn	vector	Yes	The vector used in the query.
knn	k	Yes	The number of the most similar data records returned. Important In pure vector search scenarios, we recommend that you set the size and k parameters to the same value.
ext	lvector.min_score	No	The similarity threshold. The returned vector score must be greater than this value. The returned vector score range is 0 to 1. Minimum value: [0, +inf]. Default value: `0`.
	lvector.filter_type	No	The mode of the integrated query. Valid values: pre_filter: filters structured data and then queries vector data. post_filter: queries vector data and then filters structured data. efficient_filter: automatically selects `pre_filter` or `post_filter` based on internal cost estimation. Important The version of LindormSearch must be 3.9.10 or later. This parameter is empty by default.
	lvector.ef_search	No	The length of the dynamic list during index building. This parameter is applicable only to the HNSW algorithm. Valid values: 1 to 1000. Default value: `100`.
	lvector.nprobe	No	The number of cluster units that you want to query. Adjust the value of this parameter based on your recall rate requirements to achieve the desired effect. A larger value indicates a higher recall rate and lower search performance. Minimum value: 1. Maximum value: the value specified by the method.parameters.nlist parameter. This parameter does not have a default value. Important This parameter is applicable only to the IVFPQ algorithm.
	lvector.reorder_factor	No	Creates a reordering of search results by using original vectors. The distance calculated by the IVFPQ algorithm is a quantized distance, which may result in some precision loss. Therefore, reordering by using original vectors is required. The reorder proportion is specified by the `k * reorder_factor` parameter. Reordering is used to improve the recall precision but may increase the performance overhead. Valid values: 1 to 200. Default value: `10`. Important This parameter is applicable only to the IVFPQ algorithm. If the value of the k parameter is small, set this parameter to `5`. If the value of the k parameter is greater than `100`, set this parameter to 1.
	lvector.client_refactor	No	Specifies whether to perform reordering at the upper layer of the system instead of within each shard to improve system performance. Valid values: true false (default)
	lvector. k_expand_scope	No	If you use the efficient_filter mode for approximate queries and the system automatically selects the post_filter mode, you must use the k_expand_scope parameter instead of the k parameter to achieve better performance. Default value: `1000`.

In this example, the HNSW index vector_test is used. Sample output:

Show output

{
    "took": 65,
    "timed_out": false,
    "terminated_early": false,
    "num_reduce_phases": 0,
    "_shards": {
        "total": 2,
        "successful": 2,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 10,
            "relation": "eq"
        },
        "max_score": 0.25,
        "hits": [
            {
                "_index": "vector_test",
                "_id": "1",
                "_score": 0.25
            },
            {
                "_index": "vector_test",
                "_id": "32",
                "_score": 0.14561969
            },
            {
                "_index": "vector_test",
                "_id": "122",
                "_score": 0.13761099
            },
            {
                "_index": "vector_test",
                "_id": "80",
                "_score": 0.13138853
            },
            {
                "_index": "vector_test",
                "_id": "12",
                "_score": 0.12602884
            },
            {
                "_index": "vector_test",
                "_id": "120",
                "_score": 0.123480916
            },
            {
                "_index": "vector_test",
                "_id": "39",
                "_score": 0.12126313
            },
            {
                "_index": "vector_test",
                "_id": "27",
                "_score": 0.117812514
            },
            {
                "_index": "vector_test",
                "_id": "29",
                "_score": 0.11756193
            },
            {
                "_index": "vector_test",
                "_id": "81",
                "_score": 0.11755075
            }
        ]
    }
}

Return a specified field

To return a specified field in a query, you can specify "_source": ["field1", "field2"]. To return all non-vector fields, you can specify "_source": true. The following sample code provides an example on how to query the vector_test index:

// Perform the kNN query.
Request searchRequest = new Request("GET", "/" + indexName + "/_search");
jsonString = "{"
  + "\"size\": 10,"
  + "\"_source\": [\"field1\"],"
  + "\"query\": {"
  +     "\"knn\": {"
  +         "\"vector1\": {"
  +             "\"vector\": [2.2, 2.3, 2.4],"
  +             "\"k\": 10"
  +         "}"
  +     "}"
  + "},"
  + "\"ext\": {\"lvector\": {\"min_score\": \"0.1\"}}"
  + "}";
searchRequest.setJsonEntity(jsonString);
response = restClient.performRequest(searchRequest);
responseBody = EntityUtils.toString(response.getEntity());
System.out.println("search responseBody = " + responseBody);

Sample output:

Show output

{
  "took": 31,
  "timed_out": false,
  "terminated_early": false,
  "num_reduce_phases": 0,
  "_shards": {
    "total": 2,
    "successful": 2,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 10,
      "relation": "eq"
    },
    "max_score": 0.25,
    "hits": [
      {
        "_index": "vector_test",
        "_id": "1",
        "_score": 0.25,
        "_source": {
          "field1": 1
        }
      },
      {
        "_index": "vector_test",
        "_id": "67",
        "_score": 0.15348388,
        "_source": {
          "field1": "-487556052"
        }
      },
      {
        "_index": "vector_test",
        "_id": "83",
        "_score": 0.1416535,
        "_source": {
          "field1": "1733994439"
        }
      },
      {
        "_index": "vector_test",
        "_id": "43",
        "_score": 0.13119161,
        "_source": {
          "field1": "-747555255"
        }
      },
      {
        "_index": "vector_test",
        "_id": "54",
        "_score": 0.1267109,
        "_source": {
          "field1": "-1544683361"
        }
      },
      {
        "_index": "vector_test",
        "_id": "110",
        "_score": 0.12533507,
        "_source": {
          "field1": "882740211"
        }
      },
      {
        "_index": "vector_test",
        "_id": "48",
        "_score": 0.124014825,
        "_source": {
          "field1": "-513152633"
        }
      },
      {
        "_index": "vector_test",
        "_id": "40",
        "_score": 0.12398689,
        "_source": {
          "field1": "1360426997"
        }
      },
      {
        "_index": "vector_test",
        "_id": "60",
        "_score": 0.12019993,
        "_source": {
          "field1": "10377260"
        }
      },
      {
        "_index": "vector_test",
        "_id": "61",
        "_score": 0.12009792,
        "_source": {
          "field1": "-2097991339"
        }
      }
    ]
  }
}

Use the HSNW algorithm

// Perform the kNN query.
Request searchRequest = new Request("GET", "/" + indexName + "/_search");
jsonString = "{"
  + "\"size\": 10,"
  + "\"query\": {"
  +     "\"knn\": {"
  +         "\"vector1\": {"
  +             "\"vector\": [2.2, 2.3, 2.4],"
  +             "\"k\": 10"
  +         "}"
  +     "}"
  + "},"
  + "\"ext\": {\"lvector\": {\"ef_search\": \"100\"}}"
  + "}";
searchRequest.setJsonEntity(jsonString);
response = restClient.performRequest(searchRequest);
responseBody = EntityUtils.toString(response.getEntity());
System.out.println("search responseBody = " + responseBody);

Use the IVFPQ algorithm

// Perform the kNN query.
Request searchRequest = new Request("GET", "/" + indexName + "/_search");
jsonString = "{"
  + "\"size\": 10,"
  + "\"query\": {"
  +     "\"knn\": {"
  +         "\"vector1\": {"
  +             "\"vector\": [2.2, 2.3, 2.4],"
  +             "\"k\": 10"
  +         "}"
  +     "}"
  + "},"
  + "\"ext\": {\"lvector\": {\"nprobe\": \"60\", \"reorder_factor\": \"2\"}}"
  + "}";
searchRequest.setJsonEntity(jsonString);
response = restClient.performRequest(searchRequest);
responseBody = EntityUtils.toString(response.getEntity());
System.out.println("search responseBody = " + responseBody);

Important

If the value of the k parameter is greater than 100, set the reorder_factor parameter to 1.
When the value of the nlist parameter is 10000, you can set the nprobe parameter to 60 first to view the search effect. If you want to further increase the recall rate, you can set the nprobe parameter to a larger value, such as 80, 100, 120, 140, or 160. The performance overhead incurred by increasing the value of the nprobe parameter is much less than that of adjusting the value of the reorder_factor parameter. However, we recommend that you do not set nprobe parameter to an excessively large value.

Query a sparse vector

You can use the preceding method to query a sparse vector. In this case, you must modify the format of vector1.

// Perform the kNN query.
Request searchRequest = new Request("GET", "/" + indexName + "/_search");
jsonString = "{"
  + "\"size\": 10,"
  + "\"query\": {"
  +     "\"knn\": {"
  +         "\"vector1\": {"
  +             "\"vector\": {\"indices\": [10, 45, 16], \"values\": [0.5, 0.5, 0.2]},"
  +             "\"k\": 10"
  +         "}"
  +     "}"
  + "}"
  + "}";
searchRequest.setJsonEntity(jsonString);
response = restClient.performRequest(searchRequest);
responseBody = EntityUtils.toString(response.getEntity());
System.out.println("search responseBody = " + responseBody);

Integrated query

You can integrate query conditions of vector columns and common columns to return comprehensive query results. In actual business scenarios, using the post_filter mode for approximate queries provides more similar search results.

Use a pre-filter for approximate queries

You can add a filter to the kNN query structure and set the filter_type parameter to pre_filter to filter structured data before querying vector data.

Note

The upper limit to filter structured data is 10,000 data records.

// Perform the kNN query.
Request searchRequest = new Request("GET", "/" + indexName + "/_search");
String jsonString = jsonString = "{"
  + "\"size\": 10,"
  + "\"query\": {"
  + "  \"knn\": {"
  + "    \"vector1\": {"
  + "      \"vector\": [2.2, 2.3, 2.4],"
  + "      \"filter\": {"
  + "        \"range\": {"
  + "          \"field1\": {"
  + "            \"gte\": 0"
  + "          }"
  + "        }"
  + "      },"
  + "      \"k\": 10"
  + "    }"
  + "  }"
  + "},"
  + "\"ext\": {\"lvector\": {\"filter_type\": \"pre_filter\"}}"
  + "}";
searchRequest.setJsonEntity(jsonString);
Response response = restClient.performRequest(searchRequest);
String responseBody = EntityUtils.toString(response.getEntity());
System.out.println("search responseBody = " + responseBody);

Use a post-filter for approximate queries

You can add a filter to the kNN query structure and set the filter_type parameter to post_filter to query vector data before filtering structured data.

Note

When you use the post_filter mode for approximate queries, you can set the k parameter to a larger value to query more vector data before filtering structured data.

// Perform the kNN query.
Request searchRequest = new Request("GET", "/" + indexName + "/_search");
String jsonString = "{\n" +
  "  \"size\": 10,\n" +
  "  \"query\": {\n" +
  "    \"knn\": {\n" +
  "      \"vector1\": {\n" +
  "        \"vector\": [2.2, 2.3, 2.4],\n" +
  "        \"filter\": {\n" +
  "          \"range\": {\n" +
  "            \"field1\": {\n" +
  "              \"gte\": 0\n" +
  "            }\n" +
  "          }\n" +
  "        },\n" +
  "        \"k\": 1000\n" +
  "      }\n" +
  "    }\n" +
  "  },\n" +
  "  \"ext\": {\n" +
  "    \"lvector\": {\n" +
  "      \"filter_type\": \"post_filter\"\n" +
  "    }\n" +
  "  }\n" +
  "}";
searchRequest.setJsonEntity(jsonString);
Response response = restClient.performRequest(searchRequest);
String responseBody = EntityUtils.toString(response.getEntity());
System.out.println("search responseBody = " + responseBody);

If you use the post_filter mode for approximate queries, you must increase the value of the k parameter. If you use the IVFPQ algorithm, you must adjust the value of the reorder_factor parameter. Sample code:

// Perform the kNN query.
Request searchRequest = new Request("GET", "/" + indexName + "/_search");
String jsonString = "{\n" +
  "  \"size\": 10,\n" +
  "  \"query\": {\n" +
  "    \"knn\": {\n" +
  "      \"vector1\": {\n" +
  "        \"vector\": [2.2, 2.3, 2.4],\n" +
  "        \"filter\": {\n" +
  "          \"range\": {\n" +
  "            \"field1\": {\n" +
  "              \"gte\": 0\n" +
  "            }\n" +
  "          }\n" +
  "        },\n" +
  "        \"k\": 1000\n" +
  "      }\n" +
  "    }\n" +
  "  },\n" +
  "  \"ext\": {\n" +
  "    \"lvector\": {\n" +
  "      \"filter_type\": \"post_filter\",\n" +
  "      \"nprobe\": \"60\",\n" +
  "      \"reorder_factor\": \"1\"\n" +
  "    }\n" +
  "  }\n" +
  "}";
searchRequest.setJsonEntity(jsonString);
Response response = restClient.performRequest(searchRequest);
String responseBody = EntityUtils.toString(response.getEntity());
System.out.println("search responseBody = " + responseBody);

Important

If you use the post_filter mode for approximate queries, you can set the value of the k parameter within the range of 10,000 to 20,000. This way, the latency can be shortened to below 100 milliseconds. If the value of the k parameter is large, set the reorder_factor parameter to 1.
When the value of the nlist parameter is 10000, you can set the nprobe parameter to 60 first to view the search effect. To improve the search effect, you can set the nprobe parameter to a larger value, such as 80, 100, 120, 140, or 160. The performance overhead incurred by increasing the value of the nprobe parameter is much less than that of adjusting the value of the reorder_factor parameter. However, we recommend that you do not set the nprobe parameter to an excessively large value.

You can also use the post_filter mode to add filter conditions to implement approximate queries.

// Perform the kNN query.
Request searchRequest = new Request("GET", "/" + indexName + "/_search");
String jsonString ="{\n" +
  "  \"size\": 10,\n" +
  "  \"query\": {\n" +
  "    \"knn\": {\n" +
  "      \"vector1\": {\n" +
  "        \"vector\": [2.2, 2.3, 2.4],\n" +
  "        \"k\": 10\n" +
  "      }\n" +
  "    }\n" +
  "  },\n" +
  "  \"post_filter\": {\n" +
  "    \"range\": {\n" +
  "      \"field1\": {\n" +
  "        \"gte\": 0\n" +
  "      }\n" +
  "    }\n" +
  "  }\n" +
  "}";
searchRequest.setJsonEntity(jsonString);
Response response = restClient.performRequest(searchRequest);
String responseBody = EntityUtils.toString(response.getEntity());
System.out.println("search responseBody = " + responseBody);

Common operations

Query all indexes and their data volumes.

Request request = new Request("GET", "/_cat/indices?v");
Response response = restClient.performRequest(request);
String responseBody = EntityUtils.toString(response.getEntity());
System.out.println(responseBody);

Sample output:

health status index        uuid        pri rep docs.count docs.deleted store.size pri.store.size
green  open   vector_test  vector_test 2   0          2            0      6.8kb          6.8kb

Query the data volume of a specific index.

Request request = new Request("GET", "/" + indexName + "/_count");
Response response = restClient.performRequest(request);
String responseBody = EntityUtils.toString(response.getEntity());
System.out.println(responseBody);

Sample output:

{
  "count" : 2,
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "skipped" : 0,
    "failed" : 0
  }
}

View the index creation information.

Request request = new Request("GET", "/" + indexName);
Response response = restClient.performRequest(request);
String responseBody = EntityUtils.toString(response.getEntity());
System.out.println(responseBody);

Sample output:

Show output

{
  "vector_test" : {
    "aliases" : { },
    "mappings" : {
      "_source" : {
        "excludes" : [
          "vector1"
        ]
      },
      "properties" : {
        "field1" : {
          "type" : "long"
        },
        "vector1" : {
          "type" : "knn_vector",
          "dimension" : 3,
          "data_type" : "float",
          "method" : {
            "engine" : "lvector",
            "space_type" : "l2",
            "name" : "hnsw",
            "parameters" : {
              "ef_construction" : 200,
              "m" : 24
            }
          }
        }
      }
    },
    "settings" : {
      "index" : {
        "search" : {
          "slowlog" : {
            "level" : "DEBUG",
            "threshold" : {
              "fetch" : {
                "warn" : "1s",
                "trace" : "200ms",
                "debug" : "500ms",
                "info" : "800ms"
              },
              "query" : {
                "warn" : "10s",
                "trace" : "500ms",
                "debug" : "1s",
                "info" : "5s"
              }
            }
          }
        },
        "indexing" : {
          "slowlog" : {
            "level" : "DEBUG",
            "threshold" : {
              "index" : {
                "warn" : "10s",
                "trace" : "500ms",
                "debug" : "2s",
                "info" : "5s"
              }
            }
          }
        },
        "number_of_shards" : "2",
        "provided_name" : "vector_test",
        "knn" : "true",
        "creation_date" : "1727169417350",
        "number_of_replicas" : "0",
        "uuid" : "vector_test",
        "version" : {
          "created" : "136287927"
        }
      }
    }
  }
}

Delete an entire index.

Request deleteIndexRequest = new Request("DELETE", "/" + indexName);
Response response = restClient.performRequest(deleteIndexRequest);
String responseBody = EntityUtils.toString(response.getEntity());
System.out.println("delIndex responseBody = " + responseBody);

Delete an index by query.

request = new Request("POST", "/" + indexName + "/_delete_by_query");
jsonString = "{\n" +
  "    \"query\": {\n" +
  "      \"term\": {\n" +
  "        \"field1\": 1\n" +
  "      }\n" +
  "    }\n" +
  "}";
request.setJsonEntity(jsonString);
response = restClient.performRequest(searchRequest);
responseBody = EntityUtils.toString(response.getEntity());
System.out.println("deleteByQuery responseBody = " + responseBody);