Create a search index with the Tablestore Go SDK - Tablestore

Use the `CreateSearchIndex` method to create a search index on a data table. A data table can have multiple search indexes. When you create a search index, you must add the fields that you want to query. You can also configure advanced options, such as routing fields and pre-sorting.

Prerequisites

The Tablestore client is initialized. For more information, see Initialize the Tablestore client.
You have created a data table that meets the following conditions. For more information, see Create a data table.
- The max versions must be 1.
- The time to live (TTL) is -1, or updates are disabled for the data table.

Notes

When you create a search index, the data type of a field in the search index must match the data type of the corresponding field in the data table.
If you want to set a specific TTL for the search index that is not -1, you must disable updates for the data table. The TTL of the search index must be less than or equal to the TTL of the data table. For more information, see Lifecycle management.

Parameters

When you create a search index, you must specify the table name (TableName), search index name (IndexName), and index structure (IndexSchema). The IndexSchema parameter includes FieldSchemas (settings for all index fields), IndexSetting (index settings), and IndexSort (pre-sorting settings for the index). The following table describes these parameters in detail.

Parameter	Description
TableName	The name of the data table.
IndexName	The name of the search index.
FieldSchemas	A list of FieldSchema objects. Each FieldSchema contains the following parameters: FieldName (Required): The name of the field to add to the search index. This is the column name. The type is String. Fields in a search index can be primary key columns or attribute columns. FieldType (Required): The data type of the field. The type is tablestore.FieldType_XXX. Array (Optional): Specifies whether the field is an array. The type is Boolean. If you set this parameter to true, the column is an array. When you write data, it must be in a JSON array format, such as ["a","b","c"]. Because the Nested type is an array, you do not need to set this parameter when FieldType is Nested. Index (Optional): Specifies whether to enable indexing for the field. The type is Boolean. The default value is true. This means an inverted index or a spatial index is created for the column. If you set this parameter to false, no index is created for the column. Analyzer (Optional): The tokenizer type. You can set this parameter when the field type is Text. If you do not set this parameter, the default tokenizer is single-word tokenization. EnableSortAndAgg (Optional): Specifies whether to enable sorting and statistical aggregation. The type is Boolean. You can sort results only by fields for which EnableSortAndAgg is set to true. Important Sorting and statistical aggregation are not supported for fields of the Nested type. However, they are supported for sub-columns within a Nested type field. DateFormats (Optional): The date format. The type is String. You must set this parameter when the field type is Date. For more information, see Date and time types. EnableHighlighting (Optional): Specifies whether to enable the summary and highlighting feature. The type is Boolean. The default value is false. To use the summary and highlighting feature, set this parameter to true. Only fields of the Text type support this feature. VectorOptions (Optional): The property parameters for a vector field. You must set this parameter when the field type is Vector. It includes the following parameters: DataType: The data type of the vector. Currently, only float32 is supported. If you require other types, submit a ticket to contact us. Dimension: The vector dimensions. The maximum number of dimensions supported for a vector field is 4096. MetricType: The algorithm used to measure the distance between vectors. Supported algorithms include Euclidean distance (euclidean), cosine similarity (cosine), and dot product (dot_product). euclidean: the Euclidean distance algorithm that measures the shortest path between two vectors in a multi-dimensional space. For better performance, the Euclidean distance algorithm in Tablestore does not perform the final square root calculation. A greater value that is obtained by using the Euclidean distance algorithm indicates a higher similarity between two vectors. cosine: the cosine similarity algorithm that calculates the cosine of the angle between two vectors in a vector space. A greater value that is obtained by using the cosine similarity algorithm indicates a higher similarity between two vectors. In most cases, the algorithm is used to calculate the similarity between text data. dot_product: the dot product algorithm that multiplies the corresponding coordinates of two vectors of the same dimension and adds the products. A greater value that is obtained by using the dot product algorithm indicates a higher similarity between two vectors. For information about how to select a distance measure algorithm, see Distance measure algorithms. JsonType (Optional): The index type for JSON data. Supported types are OBJECT and NESTED. You must set this parameter when the field type is JSON.
IndexSetting	Index settings. This includes the RoutingFields setting. RoutingFields (Optional): Custom routing fields. You can select some primary key columns as routing fields. Typically, you only need to set one. If you set multiple routing keys, the system concatenates their values into a single value. When writing index data, the system calculates the distribution location of the index data based on the values of the routing fields. Records with the same routing field values are indexed into the same data partition.
IndexSort	Index pre-sorting settings. This includes the Sorters setting. If you do not set this, the data is sorted by primary key by default. Note IndexSort is not supported for indexes that contain Nested type fields. These indexes do not have pre-sorting. Sorters (Required): The pre-sorting method for the index. You can sort by primary key or by field value. For more information about sorting, see Sorting and pagination. PrimaryKeySort: Sorts by primary key. It includes the following setting: Order: The sort order. You can sort in ascending or descending order. The default is ascending. FieldSort: Sorts by field value. It includes the following settings: You can pre-sort only by fields that are indexed and have sorting and statistical aggregation enabled. FieldName: The name of the field to sort by. Order: The sort order. You can sort in ascending or descending order. The default is ascending. Mode: The sorting method to use when a field has multiple values.
TimeToLive	Optional. The TTL of the data, which is the data retention period in seconds. The default value is -1, which means the data never expires. The minimum value for TTL is 86400 seconds (one day). You can also set it to -1 (never expires). When the data retention period exceeds the specified TTL, the system automatically cleans up the expired data. For more information about the search index lifecycle, see Lifecycle management.

Examples

Create a search index with default configurations

The following example shows how to create a search index. The search index contains three columns: col_keyword (Keyword type), col_long (Long type), and col_vector (Vector type).

func createSearchIndex(client *tablestore.TableStoreClient) {
    request := &tablestore.CreateSearchIndexRequest{}
    request.TableName = "<TABLE_NAME>"
    request.IndexName = "<SEARCH_INDEX_NAME>"
    request.IndexSchema = &tablestore.IndexSchema{
        FieldSchemas: []*tablestore.FieldSchema{
            {
                FieldName:        proto.String("col_keyword"),
                FieldType:        tablestore.FieldType_KEYWORD, // String type
                Index:            proto.Bool(true),
                EnableSortAndAgg: proto.Bool(true),
            },
            {
                FieldName:        proto.String("col_long"),
                FieldType:        tablestore.FieldType_LONG, // Numeric type
                Index:            proto.Bool(true),
                EnableSortAndAgg: proto.Bool(true),
            },
            {
                FieldName: proto.String("col_vector"),
                FieldType: tablestore.FieldType_VECTOR, // Vector type
                Index:     proto.Bool(true),
                VectorOptions: &tablestore.VectorOptions{
                    VectorDataType:   tablestore.VectorDataType_FLOAT_32.Enum(),
                    Dimension:        proto.Int32(4), // The vector dimension is 4, and the similarity algorithm is dot product.
                    VectorMetricType: tablestore.VectorMetricType_DOT_PRODUCT.Enum(),
                },
            },
        },
    }
    _, err := client.CreateSearchIndex(request)
    if err != nil {
        fmt.Println("Failed to create searchIndex with error:", err)
        return
    }
}

Specify IndexSort when creating a search index

The following example shows how to create a search index and specify pre-sorting for the index. The search index contains two columns: col1 (Keyword type) and col2 (Long type).

func createSearchIndex_withIndexSort(client *tablestore.TableStoreClient){
    request := &tablestore.CreateSearchIndexRequest{}
    request.TableName = "<TABLE_NAME>" // Set the table name.
    request.IndexName = "<SEARCH_INDEX_NAME>" // Set the search index name.

    schemas := []*tablestore.FieldSchema{}
    field1 := &tablestore.FieldSchema{
        FieldName: proto.String("col1"), // Set the field name. Use proto.String to get the string pointer.
        FieldType: tablestore.FieldType_KEYWORD, // Set the field type.
        Index:     proto.Bool(true), // Enable indexing.
        EnableSortAndAgg: proto.Bool(true), // Enable sorting and statistical aggregation.
    }
    field2 := &tablestore.FieldSchema{
        FieldName: proto.String("col2"),
        FieldType: tablestore.FieldType_LONG,
        Index:     proto.Bool(true),
        EnableSortAndAgg: proto.Bool(true),
    }

    schemas = append(schemas, field1, field2)
    request.IndexSchema = &tablestore.IndexSchema{
        FieldSchemas: schemas, // Set the fields to include in the search index.
        IndexSort: &search.Sort{ // Specify index pre-sorting. Sort by col2 in ascending order, then by col1 in descending order.
            Sorters: []search.Sorter{
                &search.FieldSort{
                    FieldName: "col2",
                    Order:     search.SortOrder_ASC.Enum(),
                },
                &search.FieldSort{
                    FieldName: "col1",
                    Order:     search.SortOrder_DESC.Enum(),
                },
            },
        },
    }
    resp, err := client.CreateSearchIndex(request) // Call the client to create the search index.
    if err != nil {
        fmt.Println("error :", err)
        return
    }
    fmt.Println("CreateSearchIndex finished, requestId:", resp.ResponseInfo.RequestId)
}

Set the time to live when creating a search index

Important

Make sure that updates are disabled for the data table.

func createIndexWithTTL(client *tablestore.TableStoreClient) {
    request := &tablestore.CreateSearchIndexRequest{}
    request.TableName = "<TABLE_NAME>"
    request.IndexName = "<SEARCH_INDEX_NAME>"
    schemas := []*tablestore.FieldSchema{}
    field1 := &tablestore.FieldSchema{
        FieldName:        proto.String("col1"),         // Set the field name. Use proto.String to get the string pointer.
        FieldType:        tablestore.FieldType_KEYWORD, // Set the field type.
        Index:            proto.Bool(true),             // Enable indexing.
        EnableSortAndAgg: proto.Bool(true),             // Enable sorting and statistical aggregation.
    }
    field2 := &tablestore.FieldSchema{
        FieldName:        proto.String("col2"),
        FieldType:        tablestore.FieldType_LONG,
        Index:            proto.Bool(true),
        EnableSortAndAgg: proto.Bool(true),
    }
    schemas = append(schemas, field1, field2)
    request.IndexSchema = &tablestore.IndexSchema{
        FieldSchemas: schemas, // Set the fields to include in the search index.
    }
    request.TimeToLive = proto.Int32(3600 * 24 * 7) // Set the TTL of the search index to 7 days.
    resp, err := client.CreateSearchIndex(request)
    if err != nil {
       fmt.Println("error :", err)
       return
   }
    fmt.Println("createIndexWithTTL finished, requestId:", resp.ResponseInfo.RequestId)
}

Enable query highlighting when creating a search index

The following example shows how to create a search index with four columns: col_keyword (Keyword type), col_long (Long type), col_text (Text type), and col_nested (Nested type). The col_nested column contains two sub-columns: level1_text (Text type) and level1_nested (Nested type). The level1_nested sub-column, in turn, contains the level2_text (Text type) sub-column. Query highlighting is enabled for the col_text column, the level1_text sub-column of col_nested, and the level2_text sub-column of col_nested.level1_nested.

func createSearchIndexwithHighlighting(client *tablestore.TableStoreClient) {
    request := &tablestore.CreateSearchIndexRequest{}
    request.TableName = "<TABLE_NAME>"
    request.IndexName = "<SEARCH_INDEX_NAME>"
    request.IndexSchema = &tablestore.IndexSchema{
        FieldSchemas: []*tablestore.FieldSchema{
            {
                FieldName:        proto.String("col_keyword"),
                FieldType:        tablestore.FieldType_KEYWORD, // String type.
                Index:            proto.Bool(true),
                EnableSortAndAgg: proto.Bool(true),
            },
            {
                FieldName:        proto.String("col_long"),
                FieldType:        tablestore.FieldType_LONG, // Numeric type.
                Index:            proto.Bool(true),
                EnableSortAndAgg: proto.Bool(true),
            },
            {// Enable query highlighting for a non-nested type.
                FieldName: proto.String("col_text"),
                FieldType: tablestore.FieldType_TEXT, // Tokenizable string type.
                Index:     proto.Bool(true),
                EnableSortAndAgg: proto.Bool(true),
                EnableHighlighting: proto.Bool(true),
            },
            {// Enable query highlighting for sub-columns in a nested type field.
                FieldName: proto.String("col_nested"),
                FieldType: tablestore.FieldType_NESTED,
                FieldSchemas: []*tablestore.FieldSchema{
                    {
                        FieldName:          proto.String("level1_text"),
                        FieldType:          tablestore.FieldType_TEXT,
                        Index:              proto.Bool(true),
                        EnableHighlighting: proto.Bool(true),
                    },
                    {
                        FieldName: proto.String("level1_nested"),
                        FieldType: tablestore.FieldType_NESTED,
                        FieldSchemas: []*tablestore.FieldSchema{
                            {
                                FieldName:          proto.String("level2_text"),
                                FieldType:          tablestore.FieldType_TEXT,
                                Index:              proto.Bool(true),
                                EnableHighlighting: proto.Bool(true),
                            },
                        },
                    },
                },
            },
        },
    }
    _, err := client.CreateSearchIndex(request)
    if err != nil {
        fmt.Println("Failed to create searchIndex with error:", err)
        return
    }
}

FAQ

References

After you create a search index, you can use various query types for multi-dimensional data queries. These query types include term query, terms query, match all query, match query, match phrase query, prefix query, range query, wildcard query, geo query, boolean query, AISearch, nested query, and column existence query.
When you query data, you can sort and paginate, highlight, or collapse (deduplicate) the result set.
After you create a search index, you can manage it by performing operations such as dynamically modifying the schema, managing the lifecycle, listing search indexes, querying search index descriptions, and deleting a search index.
If you want to perform data analytics, such as finding the maximum or minimum value, calculating a sum, or counting rows, you can use the statistical aggregation feature or the SQL query feature.
If you want to quickly export data and the order of the entire result set is not important, you can use the parallel scan feature.