Vector search - Tablestore SDK for Python - Tablestore - Alibaba Cloud Documentation Center

Prerequisites

Install the Tablestore SDK for Python and initialize a client.

The vector search feature requires SDK version 5.4.4 or later. We recommend that you use the latest SDK version.

A search index that contains a Vector field has been created for the data table.

Description

Vector search performs approximate nearest neighbor (ANN) calculation between a query vector and vectors in a Vector field, scores results by using the distance metric configured for the search index, and returns nearest-neighbor data. The field dimension must be the same as the query-vector dimension.

KnnVectorQuery(
    field_name,
    top_k=None,
    float32_query_vector=None,
    filter=None,
    weight=None,
    min_score=None,
    num_candidates=None,
)

The following example queries the three nearest rows to a specified four-dimensional vector and returns them in descending order of vector score.

query = KnnVectorQuery(
    "embedding",
    top_k=3,
    float32_query_vector=[1.0, 0.0, 0.0, 0.0],
)
response = client.search(
    "example_table",
    "example_index",
    SearchQuery(
        query,
        sort=Sort([ScoreSort()]),
        limit=3,
        get_total_count=False,
    ),
    ColumnsToGet(return_type=ColumnReturnType.ALL),
)
for hit in response.search_hits:
    print(hit.score, hit.row)

Important

Do not set get_total_count to True for vector search. Limits apply to the number and dimensions of vector fields and to top_k. For more information, see Search index limits.

Parameters

Search request

The search method contains the following parameters.

Name	Type	Description
table_name (required)	`str`	The name of the data table.
index_name (required)	`str`	The name of the search index.
search_query (required)	`SearchQuery`	The query condition and common query configurations.
columns_to_get (optional)	`ColumnsToGet`	The return column configuration. If this parameter is not specified, only primary key columns are returned.
routing_keys (optional)	`list`	The primary key values of custom routing fields. This parameter is not required if custom routing is not configured.
timeout_s (optional)	`int`	The request timeout in seconds. If this parameter is not specified, the client-level timeout is used.

Query configuration

search_query is of the SearchQuery type and contains the following parameters.

Name	Type	Description
query (required)	`Query`	The query condition. Set this parameter to `KnnVectorQuery`.
sort (optional)	`Sort`	The sort order of results. For more information, see Sort and paginate results.
get_total_count (optional)	`bool`	Vector search does not support total count tracking. Keep this parameter set to `False`.
next_token (optional)	`bytes`	The pagination token. Pass `next_token` from the previous response to retrieve the next page. Each server-side index partition returns its own `top_k` nearest values, which are merged by the coordinator. Therefore, when you paginate by using `next_token`, the cumulative number of rows returned depends on the number of server-side index partitions.
offset (optional)	`int`	The offset from which the query starts. Use this parameter for shallow pagination.
limit (optional)	`int`	The maximum number of rows to return. If this parameter is set to `0`, no rows are returned.
aggs (optional)	`list[Agg]`	The metric aggregation configurations. For more information, see Aggregation.
group_bys (optional)	`list[BaseGroupBy]`	The grouping configurations. For more information, see Aggregation.
collapse_field (optional)	`Collapse`	The result collapse configuration. For more information, see Collapse query results.

Vector query condition

search_query.query is of the KnnVectorQuery type and contains the following parameters.

Name	Type	Description
field_name (required)	`str`	The name of the Vector field.
top_k (required)	`int`	The number of nearest vectors to query. Maximum value: `1000`.
float32_query_vector (required)	`list[float]`	The Float32 query vector used to calculate similarity. Its length must equal the vector-field dimension.
filter (optional)	`Query`	The non-vector condition that nearest-neighbor data must also satisfy. Use `BoolQuery` to combine conditions.
weight (optional)	`float`	The vector-query weight, which must be greater than or equal to `0`. Default value: `1.0`. It affects scores but not matching.
min_score (optional)	`float`	The minimum score threshold, which must be greater than or equal to `0`. Only rows with scores strictly greater than this value are returned.
num_candidates (optional)	`int`	The number of candidates visited on each index partition. Valid values: `[top_k, 1000]`. A larger value may improve recall but increase latency.

Return columns

columns_to_get is of the ColumnsToGet type and contains the following parameters.

Name	Type	Description
column_names (optional)	`list[str]`	The names of attribute columns to return. Specify this parameter only when `return_type` is `SPECIFIED`.
return_type (optional)	`ColumnReturnType`	The return column mode. `NONE` (default) returns only primary key columns; `SPECIFIED` returns specified attribute columns; `ALL` returns all attribute columns in the table; and `ALL_FROM_INDEX` returns all stored fields in the index.

Response

The search method returns SearchResponse. The following table describes the core fields.

Field	Type	Description
rows	`list[Row]`	The rows returned by the query. The number does not exceed `limit`.
next_token	`bytes`	The token for the next page. An empty value indicates that no more data is available.
total_count	`int`	Vector search does not support total count tracking. Do not use this field.
is_all_succeed	`bool`	Indicates whether all index partitions were queried. If the value is `False`, partial results are returned.
agg_results	`list[AggResult]`	The metric aggregation results. This field is empty if `aggs` is not configured.
group_by_results	`list[GroupByResult]`	The grouping results. This field is empty if `group_bys` is not configured.
search_hits	`list[SearchHit]`	The search hits, including extended information such as rows, relevance scores, and highlights.

Tuple-compatible response

Starting from Tablestore SDK for Python 5.2.0, search APIs return response objects instead of tuples. Version 5.1.0 and earlier return tuples directly. In version 5.2.1 and later, you can call SearchResponse.v1_response() to obtain a tuple compatible with earlier versions. For new code, access SearchResponse attributes directly to avoid unpacking errors if response fields are extended.

(
    rows,
    next_token,
    total_count,
    is_all_succeed,
    agg_results,
    group_by_results,
    search_hits,
) = response.v1_response()

Examples

Filter by a non-vector condition and score

The following example returns nearest-neighbor rows whose category starts with book- and whose vector score is greater than 0.1. The top three rows are selected from 10 candidates on each index partition.

query = KnnVectorQuery(
    "embedding",
    top_k=3,
    float32_query_vector=[1.0, 0.0, 0.0, 0.0],
    filter=PrefixQuery("category", "book-"),
    min_score=0.1,
    num_candidates=10,
)
response = client.search(
    "example_table",
    "example_index",
    SearchQuery(query, limit=3, get_total_count=False),
)
for hit in response.search_hits:
    print(hit.score, hit.row)