This topic explains how to perform vector searches using the VECTOR_SEARCH_AGG function. Vector search finds semantically similar items based on a high-dimensional numerical vector. Unlike the VECTOR_SEARCH function, VECTOR_SEARCH_AGG returns all matching items as an array.
Limitations
Version support: Ververica Runtime (VVR) 11.3+ support stream mode and VVR 11.4+ support batch mode.
Vector table: Only Milvus is supported as the vector table.
Stream type: Only non-updating streams are supported (containing only
INSERTmessages).
Syntax
VECTOR_SEARCH_AGG(
TABLE <SEARCH_TABLE>,
DESCRIPTOR(<COLUMN_TO_SEARCH>),
<COLUMN_TO_QUERY>,
<TOP_K>[,
<CONFIG>]
)Input parameters
Parameter | Data type | Description |
TABLE <SEARCH_TABLE> | TABLE | The name of the vector table. |
DESCRIPTOR(<COLUMN_TO_SEARCH>) | DESC | The indexed vector column in the vector table. Input data is compared against this column to compute similarity. |
COLUMN_TO_QUERY | ARRAY<FLOAT>/ARRAY<DOUBLE> | The vector feature column from the input data, such as the embedding of an uploaded image or text. This column is matched against the indexed vector column to find similarities. |
TOP_K | INT | The maximum number of similar data entries to return. |
CONFIG | MAP<STRING,STRING> | Configurable runtime parameters. |
Return value
VECTOR_SEARCH_AGG returns a table with a single record. The table schema is ROW<ARRAY<ROW>>.
The array column is named search_results. Each ROW in the array contains all columns from the vector table and an additional column named score. The score column is of the DOUBLE data type and indicates the similarity between the input data and the output data.
Runtime parameters
Parameter | Data type | Default value | Description |
async | Boolean | (none) | Specifies whether to enable asynchronous mode. If the connector for the vector table does not support the specified mode, the engine reports an error. By default, the engine selects the execution mode based on what the connector supports. If the connector supports both asynchronous and synchronous modes, the engine prioritizes asynchronous mode to improve throughput. |
max-concurrent-operations | Integer | 10 | The maximum number of concurrent requests in asynchronous mode. |
output-mode | Enum | ORDERED | The output mode for asynchronous operations. Valid values:
For information about these values, see Async I/O. |
timeout | Duration | 3 min | The timeout for an asynchronous operation, from the first call until completion. This period can include multiple retries and is reset on failover. |
Examples
Test data
vector_table contains the following data:
id | topic | vector_index |
1 | "BigData" | [1, 1, 0] |
2 | "Streaming" | [-5, -12, -13] |
3 | "Batch" | [5, 12, 13] |
query_table contains the following data:
id | user_keyword | embedding |
1 | "Spark" | [5, 12, 13] |
2 | "Flink" | [-5, -12, -13] |
Test statement
The following SQL statement searches vector_table for each record in query_table to retrieve the top 2 most similar items.
SELECT user_keyword, search_results
FROM
query_table,
LATERAL TABLE (VECTOR_SEARCH_AGG(
SEARCH_TABLE => TABLE vector_table,
COLUMN_TO_SEARCH => DESCRIPTOR(vector_index),
COLUMN_TO_QUERY => query_table.embedding,
TOP_K => 2,
MAP['async', 'false'] -- Enable synchronous mode
))Output
user_keyword | search_results |
"Spark" | [(3, "Batch", [5.0, 12.0, 13.0], 1.0), (1, "BigData", [1.0, 1.0, 0.0], 0.6538461538461539)] |
"Flink" | [(2, "Streaming", [-5.0, -12.0, -13.0], 1.0), (1, "BigData", [1.0, 1.0, 0.0], -0.6538461538461539)] |