Elasticsearch:Vector cluster recall mode

Register a single Slice centroid

The following request registers the centroid of the Slice cluster-a:

PUT /_slice_collection/products/slices/cluster-a
{
  "vector": [0.9, 0.1]
}

A successful HTTP response does not guarantee that the centroid was created. As described in Check registration results (single and batch), check the top-level errors field and the result of every element in items.

Register Slice centroids in batches

An offline clustering job usually generates multiple vector cluster centroids at the same time. You can use the batch API to register them in one request:

PUT /_slice_collection/products/slices
{
  "slices": [
    { "slice": "cluster-b", "vector": [0.1, 0.9] },
    { "slice": "cluster-c", "vector": [0.6, 0.4] }
  ]
}

The Slice name of each item can be specified by slice or slice_id, but only one of them can appear in the same item. You can register up to 20,480 Slices in a single request, and Slice names must be unique within the request.

Check registration results (single and batch)

Both the single and the batch registration API can report item-level errors in an HTTP 200 response, including a mismatch between the centroid dimensions and config.vector_dims. A successful HTTP request does not mean that all vector clusters were registered successfully. In addition to handling non-2xx responses, you must check the top-level errors field and, for every element in items, its result: created indicates that the centroid was created, updated indicates that the centroid was updated, noop indicates that the content did not change, and failed indicates a failure. A failed item contains an error field of the string type and does not contain the backing_index field that a successful item contains.

{
  "acknowledged": true,
  "errors": false,
  "items": [
    {
      "slice_id": "cluster-b",
      "result": "created",
      "backing_index": ".sc-products-...-00000"
    },
    {
      "slice_id": "cluster-c",
      "result": "created",
      "backing_index": ".sc-products-...-00001"
    }
  ]
}

The dimensions of each centroid must match config.vector_dims. A Slice without a centroid can still be accessed explicitly, but it does not participate in automatic candidate Slice selection. When you register a Slice with the same name again, updated is returned if the centroid changed, and noop is returned if it did not change. In both cases, the documents that have already been written are not moved.

Write documents with routing_field

After you configure routing_field, the system reads the Slice from a top-level field of the document, and you no longer need to pass _slice:

PUT /products/_doc/product-1
{
  "cluster_id": "cluster-a",
  "name": "example-product",
  "embedding": [0.92, 0.08]
}

You can also explicitly provide _slice=cluster-a, and compatible APIs also accept routing=cluster-a. If you explicitly specify a Slice, the value must match the value of the document field specified by routing_field (cluster_id in this example). Otherwise, an HTTP 400 error is returned.

Update documents

After you configure routing_field, scripted updates are not supported. To modify only some fields, use the _update API with a doc request body and explicitly carry a matching _slice. If a new value must be calculated from an old value, read the old value in your application, calculate the new value, and write it back. Use if_seq_no and if_primary_term for concurrency control to prevent your update from overwriting the updates of other requests.

Write back documents after Copy or Branch

The Copy and Branch features both retain the routing_field value in a document. When you later write a document back to the destination Slice, explicitly specify the destination and make sure that the routing field in the written document matches the destination. This prevents the document from being written back to the source Slice or triggering a validation error. For more information, see Copy data semantics.

Write documents without routing_field

If routing_field is not configured, data is written in the same way as in namespace mode: every write request or Bulk item must explicitly specify _slice. For other write behaviors, such as Bulk, Refresh visibility, _update_by_query, and _delete_by_query, see Document write, read, and query operations.

Run a query

Use a query vector to automatically select candidate Slices:

POST /products/_search?query_slice_count=2
{
  "knn": {
    "field": "embedding",
    "query_vector": [0.91, 0.09],
    "k": 10,
    "num_candidates": 100
  }
}

Verify that timed_out=false and _shards.failed=0, and that hits.hits contains product-1 written in this example.

The following table describes the parameters of automatic candidate Slice selection:

A request uses only the one query vector selected by the configured path to determine the candidate Slices of the entire request. Multiple KNN clauses do not select Slices separately and then take the union of the selected Slice sets. Other KNN or hybrid query clauses are also restricted to this set of candidate Slices.

If query_slice_count is explicitly set to a non-zero value, the value must be within the range of 1 to max_query_slice_count. Otherwise, an HTTP 400 error is returned.

If the request explicitly specifies _slice or the compatible routing parameter, the query runs within the specified scope, regardless of whether those Slices have registered centroids. The centroid requirement applies only to automatic candidate Slice selection: only the Slices that have a registered centroid are considered.

When the cluster setting apack.slice_collection.search.default_to_all_slices is disabled, which is the default, an HTTP 400 error is returned if the configured path has no query vector. In this case, explicitly specify _slice. After you enable this setting, a missing query vector falls back to querying all Slices. For regular production queries, keep this setting disabled and do not rely on the all-Slice fallback. An invalid path, the use of query_vector_builder to provide the query vector, or query vector dimensions that do not match vector_dims still return an HTTP 400 error, and this setting cannot bypass those errors. For information about the setting, see Specify the query scope.

The request body of the Count API (_count) does not contain the inline query vector that automatic Slice selection requires, so the Slice cannot be selected automatically. While the all-Slice fallback stays disabled by default, you must explicitly provide _slice. Other queries that do not contain a query vector must also explicitly specify the scope.

Prewarm known candidate Slices (optional)

If you can predict which clusters will be accessed, prewarm them first to reduce the latency of the first query on cold data. In this example, only one document is written to cluster-a, so you can prewarm it once:

POST /products/_warm_slice?_slice=cluster-a&wait_for_completion=true&timeout=60s

Verify that the request succeeds and that _shards.failed=0. If _shards.total=0, first verify that the query shards are available and use an actual query to confirm that the data is searchable, and then retry the prewarm as needed. Data has already been written in this example. The synchronous wait is used only to verify this example. In a production environment, prewarm a predictable access scope asynchronously in advance and control the concurrency. Do not iterate over and prewarm all clusters before each automatic cluster selection query. For more information, see Warm Up the Slice Cache.