Tair (Redis® OSS-Compatible):Global indexes

Item

Description

Syntax

TVS.CREATEINDEXGLOBAL global_index dims algorithm distance_method [algo_param_key alog_param_value] ...

Time complexity

O(1)

Command description

Creates a global index. This command allows you to specify the specific algorithm that is used to create the index and perform queries as well as the distance function.

The global index can be deleted only by using the TVS.DELINDEXGLOBAL command.

Parameter

• global_index: the name of the global index.

• dims: the dimension of a vector. Vectors inserted into the index must have the same dimension. Valid values: 1 to 32768.

• algorithm: the algorithm used to create the index and perform queries. Valid values:

  • FLAT: uses the Flat Search algorithm to search for vectors without creating indexes. This algorithm is suitable for datasets that contain less than 10,000 records.

  • HNSW: uses HNSW graphs to create indexes and uses the HNSW algorithm to search for vectors. This algorithm is suitable for large datasets.

• distance_method: the method used to calculate the distance between two vectors. Valid values:

  • L2: the squared Euclidean distance between two vectors.

  • IP: the inner product of two vectors. You can calculate the distance between two vectors by using the following formula: 1 - Inner product of the two vectors.

  • COSINE: the cosine distance between two vectors. You can calculate the distance between two vectors by using the following formula: 1 - Cosine of the angle between the two vectors. This method involves converting the written vectors into unit vectors, which is known as L2 normalization. Therefore, the vector results may not be the same as the original vector values.

  • JACCARD: the Jaccard distance between two vectors. You can calculate the Jaccard distance between two vectors by using the following formula: 1 - Jaccard coefficient. If you set this parameter to JACCARD, you must set the data_type parameter to BINARY.

• algo_param_key and alog_param_value:

  • data_type: the vector data type. Valid values:

    • FLOAT32 (default): a four-byte single-precision floating-point number.

    • FLOAT16: a two-byte half-precision floating-point number that conforms to the IEEE 754-2008 standard. This data type reduces the storage usage of vectors, but compromises data precision. Valid values: -65519 to 65519.

    • BINARY: a binary vector that can include only 0 and 1 and supports only the Jaccard distance formula.

  • Parameters specific to HNSW indexes:

    • ef_construct: the length of the dynamic array if you use the HNSW algorithm to create a vector index. Default value: 100. Valid values: 1 to 1000. A greater value indicates higher approximate nearest neighbor (ANN) search accuracy. This results in higher performance overhead.

    • M: the maximum number of outgoing neighbors on each layer in a graph index structure. Default value: 16. Valid values: 1 to 100. A greater value indicates higher ANN search accuracy. This results in higher performance overhead.

    • auto_gc: specifies whether to enable automatic memory space reclamation for the index. Valid values: true and false. Default value: false. This parameter is available only for Tair instances whose minor version is 6.2.8.2 or later. An HNSW index uses the marked deletion method to delete vectors. After you enable this feature, the index automatically reclaims memory space, which helps reduce memory usage. However, this feature can affect the index performance of nearest-neighbor queries. Therefore, we recommend that your enable this feature only in scenarios where memory usage is a concern, and a decrease in nearest-neighbor query performance can be tolerated.

  • Parameters specific to hybrid indexes:

    • If you want to create an inverted index for a specific tag field, you must specify the field name and the corresponding data type in advance.

      The inverted_index_<field_name> int|long|float|double|string syntax is used to create an inverted index for the field specified by field_name. The supported data types are INT, LONG, FlOAT, DOUBLE, and STRING. The field name and the corresponding data type must be in lowercase.

      For example, if you want to create an inverted index on a field called productname of the STRING type, the syntax is inverted_index_productname string.

    • lexical_algorithm: the full-text search algorithm. Valid values:

      • bm25: the Okapi BM25 algorithm. You can pass in the original text and use the Tair vector service to create an index.

      • vector: the vector algorithm. You must encode the original text into sparse vectors and provide them as input to the vector algorithm. Specify the input in the "[[2,0.221],[42,09688],..]" format. The Key field specifies an index and must be of the uint32_t type, and the Value field specifies the lexical frequency profile and must be of the FLOAT type.

    • If you set the lexical_algorithm parameter to bm25, you can further customize the behavior of the BM25 algorithm by adjusting the following parameters:

      • analyzer: the analyzer that you want to use for text segmentation. Valid values: jieba and ik_smart. Default value: jieba.

      • k1: the term frequency saturation in the BM25 algorithm. The value must be greater than 0. Default value: 1.2.

      • b: the impact of the document length on the relevance score in the BM25 algorithm. Valid values: 0 to 1. Default value: 0.75.

    • hybrid_ratio: the default weight assigned to vector search during the query. Valid values: 0 to 1. Default value: 0.5. The value must be of the FLOAT type. The weight assigned to full-text search in the query results is equal to 1 minus the hybrid_ratio value.

Output

• If the operation is successful, OK is returned.

• Otherwise, an error message is returned.

Example

Sample commands:

# Create a vector index that uses the HNSW algorithm and the Jaccard distance formula for vectors whose dimension is 2 and data type is BINARY. 
TVS.CREATEINDEXGLOBAL index_name0 2 HNSW JACCARD data_type BINARY

# Create a vector index that uses the HNSW algorithm and the Euclidean distance formula for vectors whose dimension is 2 and data type is FLOAT32. 
TVS.CREATEINDEXGLOBAL index_name1 2 HNSW L2 

# Create a vector index that uses the Flat Search algorithm and the inner product formula for vectors whose dimension is 2 and data type is FLOAT32. 
TVS.CREATEINDEXGLOBAL index_name2 2 FLAT IP 

# Create a vector index that uses the Flat Search algorithm and the Jaccard distance formula for vectors whose dimension is 2 and data type is BINARY. 
TVS.CREATEINDEXGLOBAL index_name3 2 FLAT JACCARD data_type BINARY

# Create a vector index that uses the HNSW algorithm, the BM25 full-text search algorithm, and the inner product formula for vectors whose dimension is 2 and data type is FLOAT32. Specify the productname field of the STRING type as an inverted index. 
TVS.CREATEINDEXGLOBAL index_name4 2 HNSW IP lexical_algorithm bm25 inverted_index_productname string

Sample output:

OK

Item

Description

Syntax

TVS.EXPANDINDEXGLOBAL global_index num

Time complexity

O(1)

Command description

Creates data indexes in a global index.

When you specify the number of data indexes that you want to create and run this command, Tair automatically and evenly distributes these data indexes across different shards and assigns names to the data indexes. The metadata of the data indexes is the same as that of the global index. Although the data indexes are associated with the global index, they are stored independently in Tair.

Parameter

• global_index: the name of the global index.

• num: the number of data indexes that you want to create.

Output

• If the operation is successful, OK is returned.

• Otherwise, an error message is returned.

Example

Sample command:

TVS.EXPANDINDEXGLOBAL index_name0 3

Sample output:

OK

Item

Description

Syntax

TVS.IMPORTINDEXGLOBAL global_index sub_index

Time complexity

O(1)

Command description

Associates an existing data index with a global index. The metadata of the existing data index must be the same as that of the global index.

Parameter

• global_index: the name of the global index.

• sub_index: the name of the data index.

Output

• If the operation is successful, OK is returned.

• If the metadata of the data index is different from that of the global index, the (error)ERR tow indices are not suitable error is returned.

• Otherwise, an error message is returned.

Example

Sample command:

TVS.IMPORTINDEXGLOBAL index_name0 testvector

Sample output:

OK

Item

Description

Syntax

TVS.LISTINDEXGLOBAL num [global_index] [global_index] ...

Time complexity

O(1)

Command description

Queries the information about global indexes.

Parameter

• num: the number of global indexes that you want to query. If this parameter is set to 0, the information about all global indexes is returned.

• global_index: the name of the global index.

Output

• If the operation is successful, OK is returned.

• Otherwise, an error message is returned.

Example

Sample command:

TVS.LISTINDEXGLOBAL 2 index_name0 index_name4

Sample output:

1) "{\"dimension\":\"2\",\"algorithm\":\"HNSW\",\"distance_method\":\"JACCARD\",\"parameters\":[\"data_type\",\"BINARY\"],\"sub_index_list\":[\"index_name0_sub_1719912517_2\",\"index_name0_sub_1719912517_3\",\"index_name0_sub_1719912517_4\"]}"
2) "{\"dimension\":\"2\",\"algorithm\":\"HNSW\",\"distance_method\":\"IP\",\"parameters\":[\"lexical_algorithm\",\"bm25\",\"inverted_index_productname\",\"string\"],\"sub_index_list\":[\"index_name4_sub_1719986971_2\",\"index_name4_sub_1719986971_0\",\"index_name4_sub_1719986971_1\"]}"

Item

Description

Syntax

TVS.DELINDEXGLOBAL global_index [sub_index] ...

Time complexity

O(1)

Command description

Disassociates a global index from its associated data indexes. If no data index is specified, the global index is deleted and all related data indexes are disassociated from the global index.

Parameter

• global_index: the name of the global index.

• sub_index: the name of the data index.

Output

• If the operation is successful, the return value varies based on whether the global index is deleted.

  • If the command dissociates the global index from its associated data indexes without deleting the global index, a value of 0 is returned.

  • If the command deletes the global index, a value of 1 is returned.

• Otherwise, an error message is returned.

Example

Sample command:

TVS.DELINDEXGLOBAL index_name0 index_name0_sub_1720000607_0 index_name0_sub_1720000607_1

Sample output:

(integer) 0

Item

Description

Syntax

TVS.HSETGLOBAL global_index key attribute_key attribute_value [attribute_key attribute_value] ...

Time complexity

If the operation of inserting or updating data does not involve the creation or update of a vector value, the time complexity is O(1). Otherwise, the time complexity is O(log(N)), where N indicates the number of keys in the vector index.

Command description

Inserts keys into a global index. After keys are inserted, Tair automatically writes the keys to the appropriate data indexes to ensure that data is evenly balanced across all data indexes.

If a key already exists, its value is updated with the new value.

Parameter

• global_index: the name of the global index.

• key: the primary key identifier of the vector, which can be deleted by using the TVS.DELGLOBAL command.

• attribute_key and attribute_value: the vector that you want to insert in the key-value format.

  • To insert a vector, set attribute_key to VECTOR and attribute_value to a vector string that has the same dimension as that of the vector index specified by the dims parameter. Example: VECTOR [1,2]. You can write only one vector to a key. If you write a vector more than once, the existing data is overwritten.

  • To insert text data into a hybrid index that has required parameters specified, set attribute_key to TEXT and attribute_value to text data such as "TairVector is an in-house vector database service of Tair" or vectorized data such as "[[2,0.221],[42,09688],...]".

  • To insert other attributes, specify custom parameters related to these attributes. Examples: create_time 1663637425 and location Hangzhou.

Output

• If the operation is successful, the number of inserted keys is returned. If an existing field is updated, a value of 0 is returned.

• Otherwise, an error message is returned.

Example

Sample command:

TVS.HSETGLOBAL my_index key0 VECTOR [5,6] creation_time 1750 productname "Aliyun"

Sample output:

(integer) 3

Item

Description

Syntax

TVS.HGETALLGLOBAL global_index key

Time complexity

O(1)*M, where M indicates the number of data indexes.

Command description

Queries all data records of a key in a global index.

Parameter

• global_index: the name of the global index.

• key: the primary key identifier of the vector.

Output

• If the operation is successful, all data records of the key are returned.

• If the specified vector index or key does not exist, (empty array) is returned.

• Otherwise, an error message is returned.

Example

Sample command:

TVS.HGETALLGLOBAL my_index key0

Sample output:

1) "VECTOR"
2) "[5,6]"
3) "productname"
4) "Aliyun"
5) "creation_time"
6) "1750"

Item

Description

Syntax

TVS.HMGETGLOBAL global_index key attribute_key [attribute_key ...]

Time complexity

O(1)*M, where M indicates the number of data indexes.

Command description

Queries the values of attribute keys in the specified key of the specified vector index.

Parameter

• global_index: the name of the global index.

• key: the primary key identifier of the vector.

• attribute_key: the attribute key that you want to manage. You can specify multiple keys. To query a specific vector, set attribute_key to VECTOR. To query the raw text data in the full-text index, set attribute_key to TEXT.

Output

• If the operation is successful, the values of the attribute keys are returned.

• Otherwise, an error message is returned.

Example

Sample command:

TVS.HMGETGLOBAL index_name0 key0 create_time VECTOR

Sample output:

1) "1800"
2) "[7,8]"

Item

Description

Syntax

TVS.DELGLOBAL global_index key [key ...]

Time complexity

O(1)

Command description

Deletes specific keys from a global index.

Parameter

• global_index: the name of the global index.

• key: the primary key identifier of the vector. You can specify multiple keys.

Output

• If the operation is successful, the specified keys are deleted and the number of deleted keys is returned.

• If the specified index does not exist, a value of 0 is returned.

• Otherwise, an error message is returned.

Example

Sample command:

TVS.DELGLOBAL my_index key0 key1

Sample output:

 (integer) 2

Item

Description

Syntax

TVS.HDELGLOBAL global_index key attribute_key [attribute_key ...]

Time complexity

O(1)

Command description

Deletes the specified attribute key and its value from the specified key of a global index.

Parameter

• global_index: the name of the global index.

• key: the primary key identifier of the vector. You can specify multiple keys.

• attribute_key: the attribute key that you want to manage. You can specify multiple attribute keys. To delete vector data, set attribute_key to VECTOR. To delete full index data, set attribute_key to TEXT.

Output

• If the operation is successful, the specified attributes are deleted and the number of deleted attributes is returned.

• If the specified index does not exist, a value of 0 is returned.

• Otherwise, an error message is returned.

Example

Sample command:

TVS.HDELGLOBAL index_name0 keyc VECTOR

Sample output:

 (integer) 1

Item

Description

Syntax

TVS.SCANGLOBAL global_index cursor [MATCH pattern] [COUNT count] [FILTER filter_string] [VECTOR vector] [MAX_DIST max_distance]

Time complexity

O(N), where N indicates the number of keys in a data index.

Command description

Scans the keys that meet specific criteria in a global index.

Tair returns the results of only one data shard in each scan.

Parameter

• global_index: the name of the global index.

• cursor: the cursor used in this scan. The cursor position starts from 0.

• pattern: the pattern used in this scan.

• count: the number of keys that you want to scan. Default value: 10. The system may not always return the precise number of keys specified by the count parameter in each iteration.

• filter_string: the filter conditions.

  • Operators such as + - * / < > ! = ( ) && || are supported. You cannot compare the sizes of strings. If you want to enter a string, you must add escape characters (\). Example: create_time > 1663637425 && location == \"Hangzhou\".

  • You must add spaces to both sides of an operator. Example: "creation_time > 1735".

  • The flag == true format that uses the BOOLEAN data type is not supported. You can use the flag == \"true\" format, which passes the boolean value as a string.

• vector: the query vector. This parameter must be used together with the max_distance parameter.

• max_distance: the maximum distance allowed. This parameter must be used together with the vector parameter. After you specify these two parameters, the returned array includes all vectors that are within the distance specified by the max_distance parameter from the query vector.

Output

• If the operation is successful, an array is returned.

  • Element 1 of the array: the cursor used for the next scan. If the scan is complete, a value of 0 is returned.

  • Element 2 of the array: the names of scanned keys.

• Otherwise, an error message is returned.

Example

Sample command:

TVS.SCANGLOBAL index_name0 0

Sample output:

1) "72057594037927936"
2) 1) "key2"
   2) "key1"

Item

Description

Syntax

TVS.KNNSEARCHGLOBAL global_index topN vector [filter_string] [param_key param_value]

Time complexity

• HNSW algorithm: O(log(N))*M

• FLAT algorithm: O(N)*M

N indicates the number of keys in a data index, and M indicates the number of data indexes.

Command description

Queries the nearest-neighbor vectors of the specified vector in a global index. Up to 10,000 vectors can be returned.

For example, if you want to query the topN value, Tair retrieves the top N vectors from each data index. In this case, the maximum number of vectors is the topN value multiplied by the number of data indexes. After the top N vectors are retrieved from each data index, Tair aggregates and sorts all these results and returns the final topN value.

Parameter

• global_index: the name of the global index.

• topN: the number of vectors to return. Valid values: 1 to 10000.

• vector: the vector whose nearest neighbors you want to query. If you want to perform only full-text search by using the hybrid indexing method, you can set the vector field to an empty string ("").

• filter_string: the filter conditions.

  • Operators such as + - * / < > ! = ( ) && || are supported. You cannot compare the sizes of strings. If you want to enter a string, you must add escape characters (\). Example: "create_time > 1663637425 && location == \"Hangzhou\"".

  • You must add spaces to both sides of an operator. Example: "creation_time > 1735".

  • The flag == true format that uses the BOOLEAN data type is not supported. Instead, you can use the flag == \"true\" format, which passes the boolean value as a string.

• param_key and param_value: the operational parameters of the query. Valid values:

  • ef_search: the length of the dynamic array. Default value: 100. Valid values: 1 to 1000. A greater value indicates higher search accuracy. This results in higher performance overhead. This parameter is available only if you use the HNSW algorithm.

  • MAX_DIST: the maximum distance allowed, which must be of the FLOAT type. If the distance between the specified vector and a neighbor vector is greater than the value of this parameter, the key of the neighbor vector is filtered out and excluded from the search results.

  • TEXT: the text on which you want to perform a hybrid search. You can pass text or vector data to this field. If you do not specify this field or if you pass an empty string ("") to this field, only vector search is performed.

  • hybrid_ratio: the weight assigned to vector search during the query. By default, this field takes the value specified in the hybrid_ratio parameter of the TVS.CREATEINDEX command. Valid values: 0 to 1. The value must be of the FLOAT type. The weight assigned to full-text search in the query results is equal to 1 minus the hybrid_ratio value.

  • By default, the system performs k-nearest neighbors (KNN) vector search followed by scalar filtering, which is known as the PostFilter policy.

    • vector_filter_count: the maximum number of records that can be filtered during a vector search. Default value: 10000.

      In the PostFilter policy, if the number of records filtered from the vector search results exceeds the vector_filter_count value, but the operation still cannot gather a sufficient amount of data to return, the system terminates the vector search.

    • fulltext_filter_count: the maximum number of records that can be filtered during a full-text search. Default value: 10000.

      In the PostFilter policy, if the number of records filtered from the full-text search results exceeds the fulltext_filter_count value, but the operation still cannot gather a sufficient amount of data to return, the system terminates the full-text search.

    • If you want the system to perform scalar filtering based on inverted indexes before performing KNN vector search, you can use the PreFilter policy by adding the search_policy scala parameter to the query.

      The ivf_filter_count parameter is also available to specify the maximum number of records that can be filtered based on inverted indexes. The default value is 10000. In the PreFilter policy, if the number of records processed during this initial scalar filtering stage exceeds the ivf_filter_count value, the system automatically degrades to the PostFilter policy.

    Note

    In most cases, the default settings of the preceding parameters can ensure that the system operates with acceptable latency while maintaining accuracy. You can adjust these parameters on a per-request basis to better fit the specific needs of your use case. However, increasing the values of the preceding parameters may lead to higher query latency.

Output

• If the operation is successful, the keys of the neighbor vectors are returned in ascending order of their distances to the specified vector.

• If the specified vector indexes do not exist, (empty array) is returned.

• Otherwise, an error message is returned.

Example

TVS.CREATEINDEXGLOBAL my_global_index 2 HNSW L2 inverted_index_productname string
TVS.EXPANDINDEXGLOBAL my_global_index 3
TVS.HSETGLOBAL my_global_index key0 VECTOR [1,2] creation_time 1730 productname "Aliyun"
TVS.HSETGLOBAL my_global_index key1 VECTOR [3,4] creation_time 1740 productname "other"
TVS.HSETGLOBAL my_global_index key2 VECTOR [5,6] creation_time 1750 productname "Aliyun"

Sample command 1:

TVS.KNNSEARCHGLOBAL my_global_index 2 [3,3.1] "creation_time > 1735"

Sample output 1:

1) "key1"
2) "0.81000018119812012"
3) "key2"
4) "12.410000801086426"

Sample command 2:

TVS.KNNSEARCHGLOBAL my_global_index 2 [3,3.1]  "creation_time > 1735 && productname ==  \"Aliyun\"" search_policy scala ivf_filter_count 15000

Sample output 2:

1) "key2"
2) "12.410000801086426"

Item

Description

Syntax

TVS.KNNSEARCHFIELDGLOBAL global_index topN vector field_count field_name [field_name ...] [filter_string] [param_key param_value]

Time complexity

• HNSW algorithm: O(log(N))*M

• FLAT algorithm: O(N)*M

N indicates the number of keys in a data index, and M indicates the number of data indexes.

Command description

Queries the nearest-neighbor vectors of the specified vector in a global index. The search logic of this command is the same as that of TVS.KNNSEARCHGLOBAL, with the additional capability to return tag attributes along with the search results.

Parameter

• global_index: the name of the global index.

• topN: the number of vectors to return. Valid values: 1 to 10000.

• vector: the vector whose nearest neighbors you want to query. If you want to perform only full-text search by using the hybrid indexing method, you can set the vector field to an empty string ("").

• field_count: the number of tag attributes to return. If you want to return all tags, set the field_count parameter to 0.

• field_name: the tag name. The number of tags must be the same as the field_count value.

• filter_string: the filter conditions. For more information, see the description of TVS.KNNSEARCHGLOBAL.

• param_key and param_value: the operational parameters of the query. For more information, see the description of TVS.KNNSEARCHGLOBAL.

Output

• If the operation is successful, the keys of the neighbor vectors are returned in ascending order of their distances to the specified vector. The tag attributes of the vectors are also returned in the form of key-value pairs.

• If the specified vector index does not exist, (empty array) is returned.

• Otherwise, an error message is returned.

Example

TVS.CREATEINDEXGLOBAL my_global_index 2 HNSW L2 inverted_index_productname string
TVS.EXPANDINDEXGLOBAL my_global_index 3
TVS.HSETGLOBAL my_global_index key0 VECTOR [1,2] creation_time 1730 productname "Aliyun"
TVS.HSETGLOBAL my_global_index key1 VECTOR [3,4] creation_time 1740 productname "other"
TVS.HSETGLOBAL my_global_index key2 VECTOR [5,6] creation_time 1750 productname "Aliyun"

Sample command:

TVS.KNNSEARCHFIELDGLOBAL my_global_index 2 [3,3.1] 1 VECTOR "creation_time > 1735"

Sample output:

1) 1) "key1"
   2) "0.81000018119812012"
   3) "VECTOR"
   4) "[3,4]"
2) 1) "key2"
   2) "12.410000801086426"
   3) "VECTOR"
   4) "[5,6]"