Create a search index - Tablestore SDK for Python - Tablestore

Use Tablestore SDK for Python to create a search index and configure index fields, presorting, time to live (TTL), virtual columns, and highlighting.

Prerequisites

Install the Tablestore SDK for Python and initialize a client.
A table with the maximum number of versions set to 1
The table TTL set to -1 or updates by UpdateRow disabled for the table

Description

Call create_search_index to create a search index for a table. A table can have multiple search indexes. Specify the table, index name, and complete index configuration in the request, and add the columns to query to fields. Index field types must match the data types of the corresponding table columns. For supported types, see Data types.

create_search_index(table_name, index_name, index_meta)

Note

Search index creation is asynchronous. After the request succeeds, wait for index data synchronization to complete before querying the index.

The following example creates example_index on example_table with Keyword and Long fields. If presorting and TTL are not configured, the index is sorted by primary key and index data does not expire.

table_name = "example_table"
index_name = "example_index"

fields = [
    FieldSchema("category", FieldType.KEYWORD, index=True),
    FieldSchema("price", FieldType.LONG, index=True),
]
index_meta = SearchIndexMeta(fields)

client.create_search_index(table_name, index_name, index_meta)

Parameters

create_search_index provides the following parameters.

Name	Type	Description
table_name (required)	`str`	The table name.
index_name (required)	`str`	The search index name.
index_meta (required)	`SearchIndexMeta`	The search index configuration.

Index configuration

index_meta is of the SearchIndexMeta type and provides the following parameters.

Name	Type	Description
fields (required)	`List[FieldSchema]`	The index fields.
index_setting (optional)	`IndexSetting`	The index settings.
index_sort (optional)	`Sort`	The presorting configuration. If this parameter is not configured and the index does not contain a Nested field, data is sorted by primary key. Nested indexes do not support presorting.
time_to_live (optional)	`int`	The TTL of index data, in seconds. Default value: -1. The value must be -1 or an integer greater than or equal to 86400. A value of -1 specifies that index data does not expire. If you specify another value, disable updates by `UpdateRow` for the table. The index TTL cannot exceed the table TTL. You can modify this value with `update_search_index` after the index is created. For more information, see Lifecycle management.

Index fields

Each element in index_meta.fields[] is of the FieldSchema type and provides the following parameters.

Name	Type	Description
field_name (required)	`str`	The index field name. The field can correspond to a primary key column or an attribute column.
field_type (required)	`FieldType`	The index field type. Use Nested for multilayer logical relationships, JSON for JSON-formatted data, and Geo-point for geographic coordinates.
index (optional)	`bool`	Specifies whether to create an inverted or spatial index for the field. Default value: `False`. Set this parameter to `True` to query the field.
enable_highlighting (optional)	`bool`	Specifies whether to enable summary and highlighting. Only Text fields support this feature. Default value: `False`. This parameter requires Tablestore SDK for Python 6.0.0 or later. We recommend that you use the latest version.
analyzer (optional)	`str`	The analyzer for a Text field. If this parameter is not configured, single-word tokenization is used.
analyzer_parameter (optional)	`AnalyzerParameter`	The analyzer parameters. If `analyzer` is configured, specify parameters for the selected analyzer.
enable_sort_and_agg (optional)	`bool`	Specifies whether to enable sorting and aggregation. Default value: `False`. Text and Nested fields do not support sorting or aggregation, but child fields of a Nested field do.
is_array (optional)	`bool`	Specifies whether the field is an array. Default value: `False`. Array values must be written as JSON arrays. You do not need to configure this parameter for Nested fields.
sub_field_schemas (optional)	`List[FieldSchema]`	The child fields of a Nested or JSON field. This parameter is required for Nested and JSON fields.
is_virtual_field (optional)	`bool`	Specifies whether the field is a virtual column. Default value: `False`.
source_fields (optional)	`List[str]`	The source table fields mapped to the virtual column. This parameter is required for a virtual column. Only one source field is supported.
date_formats (optional)	`List[str]`	The date formats supported by a Date field. This parameter is required for Date fields.
vector_options (optional)	`VectorOptions`	The vector data type, dimension, and distance metric for a Vector field. This parameter is required for Vector fields.
json_type (optional)	`JsonType`	The JSON index type. Valid values: `JsonType.OBJECT_JSON` and `JsonType.NESTED_JSON`. This parameter is required for JSON fields.
text_similarity (optional)	`TextSimilarity`	The similarity algorithm for a Text field. Valid values: `TextSimilarity.BM25` and `TextSimilarity.SHORT_TEXT`.

Vector configuration

index_meta.fields[].vector_options is of the VectorOptions type and provides the following parameters.

Name	Type	Description
data_type (required)	`VectorDataType`	The vector data type. Only `VectorDataType.VD_FLOAT_32` is supported.
dimension (required)	`int`	The vector dimension. Maximum value: 4096.
metric_type (required)	`VectorMetricType`	The distance metric. Valid values: `VectorMetricType.VM_EUCLIDEAN`, `VectorMetricType.VM_COSINE`, and `VectorMetricType.VM_DOT_PRODUCT`. For more information, see Vector search.

Index settings

index_meta.index_setting is of the IndexSetting type and provides the following parameter.

Name	Type	Description
routing_fields (optional)	`List[str]`	The custom routing fields. You can specify one or more primary key columns. In most cases, specify one column. If you specify multiple columns, their values are concatenated. Records with the same routing field value are written to the same data partition.

Presorting configuration

index_meta.index_sort is of the Sort type and provides the following parameter.

Name	Type	Description
sorters (required)	`List[Sorter]`	The presorting rules. `PrimaryKeySort` sorts by primary key, and `FieldSort` sorts by field value. A field used by `FieldSort` must have sorting and aggregation enabled. For more information, see Sorting and paging.

Primary key sorting

An element of the PrimaryKeySort type in index_meta.index_sort.sorters[] sorts data by primary key and provides the following parameter.

Name	Type	Description
sort_order (optional)	`SortOrder`	The sort order. Valid values: `SortOrder.ASC` and `SortOrder.DESC`. Default value: `SortOrder.ASC`.

Field value sorting

An element of the FieldSort type in index_meta.index_sort.sorters[] sorts data by field value and provides the following parameters.

Name	Type	Description
field_name (required)	`str`	The field name.
sort_order (optional)	`SortOrder`	The sort order. Valid values: `SortOrder.ASC` and `SortOrder.DESC`. Default value: `SortOrder.ASC`.
sort_mode (optional)	`SortMode`	The value used for sorting when a field contains multiple values.

Examples

Configure presorting

The following example presorts index data by the created_at field in ascending order.

table_name = "example_table"
index_name = "example_index"

fields = [
    FieldSchema("category", FieldType.KEYWORD, index=True),
    FieldSchema(
        "created_at",
        FieldType.LONG,
        index=True,
        enable_sort_and_agg=True,
    ),
]
index_sort = Sort([FieldSort("created_at", SortOrder.ASC)])
index_meta = SearchIndexMeta(fields, index_sort=index_sort)

client.create_search_index(table_name, index_name, index_meta)

Configure the index TTL

The following example sets the index TTL to 7 days.

table_name = "example_table"
index_name = "example_index"

fields = [
    FieldSchema("category", FieldType.KEYWORD, index=True),
    FieldSchema("price", FieldType.LONG, index=True),
]
index_meta = SearchIndexMeta(fields, time_to_live=7 * 24 * 60 * 60)

client.create_search_index(table_name, index_name, index_meta)

Configure tokenization

The following example configures split tokenization for the Text field description and uses a comma (,) as the delimiter.

table_name = "example_table"
index_name = "example_index"

fields = [
    FieldSchema("category", FieldType.KEYWORD, index=True),
    FieldSchema(
        "description",
        FieldType.TEXT,
        index=True,
        analyzer=AnalyzerType.SPLIT,
        analyzer_parameter=SplitAnalyzerParameter(","),
    ),
]
index_meta = SearchIndexMeta(fields)

client.create_search_index(table_name, index_name, index_meta)

Create a vector field

The following example creates a four-dimensional Vector field named embedding and uses the dot product to measure vector similarity.

table_name = "example_table"
index_name = "example_index"

fields = [
    FieldSchema("category", FieldType.KEYWORD, index=True),
    FieldSchema(
        "embedding",
        FieldType.VECTOR,
        index=True,
        vector_options=VectorOptions(
            data_type=VectorDataType.VD_FLOAT_32,
            dimension=4,
            metric_type=VectorMetricType.VM_DOT_PRODUCT,
        ),
    ),
]
index_meta = SearchIndexMeta(fields)

client.create_search_index(table_name, index_name, index_meta)

Create virtual columns

The following example maps the Keyword field category to a Long virtual column and the Long field price to a Keyword virtual column.

table_name = "example_table"
index_name = "example_index"

fields = [
    FieldSchema("category", FieldType.KEYWORD, index=True),
    FieldSchema(
        "category_as_long",
        FieldType.LONG,
        index=True,
        is_virtual_field=True,
        source_fields=["category"],
    ),
    FieldSchema("price", FieldType.LONG, index=True),
    FieldSchema(
        "price_as_keyword",
        FieldType.KEYWORD,
        index=True,
        is_virtual_field=True,
        source_fields=["price"],
    ),
]
index_meta = SearchIndexMeta(fields)

client.create_search_index(table_name, index_name, index_meta)

Enable highlighting

The following example enables highlighting for the Text field description.

table_name = "example_table"
index_name = "example_index"

fields = [
    FieldSchema("category", FieldType.KEYWORD, index=True),
    FieldSchema("price", FieldType.LONG, index=True),
    FieldSchema(
        "description",
        FieldType.TEXT,
        index=True,
        enable_highlighting=True,
    ),
]
index_meta = SearchIndexMeta(fields)

client.create_search_index(table_name, index_name, index_meta)