OpenSearch:Index table configuration

Introduction to index table configuration

​

Index table configuration is a critical configuration in Retrieval Engine Edition. It defines the data format of source documents and how to build indexes, including inverted, forward, and summary indexes.

​

Index table configuration

Configuration overview

{
    "table_name":"sample",
    "fields":[

    ],
    "indexs":[

    ],
    "attributes":[

    ],
    "summarys":{

    },
    "dictionaries":[

    ],
          "adaptive_dictionaries":[
      
    ],
    "enable_ttl":true,
    "ttl_field_name":"ttl_filed",
    "default_ttl":86400
}

• table_name: The name of the index table. The system uses this name to generate an index table configuration file named $table_name_schema.json.

• fields: The fields used for index building.

• indexes: The inverted index configuration.

• attributes: The forward index configuration.

• summarys: The summary index configuration.

• dictionaries: The dictionaries for building bitmap indexes. Not required if you do not use bitmap indexes. High-frequency or adaptive dictionaries can reduce index space and improve retrieval performance.

• adaptive_dictionaries: The configuration for building adaptive bitmap indexes. Generates high-frequency words and corresponding bitmap indexes based on configured rules. Omit this item if not needed.

• file_compress: Parameters and aliases for file compression methods. Forward, inverted, and summary indexes support file-level compression.

• enable_ttl: Specifies whether to enable the time to live (TTL) feature, which automatically deletes expired data. Default value: false.

• ttl_field_name: The table field to use as the TTL field. If not configured, the built-in field ops_doc_time_to_live_in_seconds is used. If a source document does not provide a value for the TTL field, the default TTL value applies. The field must be a single-value uint32 field.

• default_ttl: The default TTL value. If you set enable_ttl to true but do not configure default_ttl, the default value is std::numeric_limits<int64_t>::max() >> 20. If you configure default_ttl, enable_ttl is automatically set to true.

​

Fields configuration

 "fileds":[
        {
            "field_name":"title",
            "field_type":"TEXT",
            "analyzer":"chn_standard"
        },
        {
            "field_name":"dup_title",
            "field_type":"TEXT",
            "analyzer":"fuzzy"
            "user_defined_param" : {
                "copy_from" : "title"
            }
        },
        {
            "field_name":"category",
            "field_type":"INTEGER",
            "multi_value":true,
            "compress_type":"uniq|equal"
        },
        {
            "field_name":"mlr_features",
            "field_type":"INTEGER",
            "multi_value":true,
            "updatable_multi_value":true
        },
        {
            "field_name":"feature",
            "field_type":"float",
            "multi_value":true,
            "fixed_multi_value_count":32,
            "compress_type":"uniq|fp16",
            "updatable_multi_value":true
        },
        {
            "field_name":"user_id",
            "field_type":"INTEGER"
        },
        {
            "field_name":"price",
            "field_type":"INTEGER",
            "enable_null":true,
            "default_null_string":"default_null"
        },
        {
            "field_name":"product_id",
            "field_type":"LONG"
        },
        {
            "field_name":"product_type",
            "field_type":"UINT8",
            "compress_type":"equal"
        },
        {
            "field_name":"bitwords",
            "field_type":"STRING",
            "multi_value":true
        },
        {
            "field_name":"date",
            "field_type":"DATE"
        },
        {
            "field_name":"time",
            "field_type":"TIME"
        },
        {
            "field_name":"timestamp",
            "field_type":"TIMESTAMP",
            "default_time_zone":"+0800"
        }
    ]

• field_name: The name of the field.

• field_type: The data type of the field. For more information, see Built-in field types in Retrieval Engine Edition.

• analyzer: The analyzer for TEXT fields. Required for TEXT fields and not configurable for other field types. For more information about built-in analyzers, see Analyzers. To apply different analyzers to a single field, add a new field in the schema and configure user_defined_param. See the dup_title field for an example.

• multi_value: Specifies whether the field is a multi-value field. If used to build an attribute, a multi-value attribute is created. Default value: false.

• updatable_multi_value: Specifies whether this multi-value or single-value STRING field can be updated. Default value: false. Supported types: INT8, UINT8, INT16, UINT16, INTEGER (32-bit integer), UINT32, LONG (64-bit integer), UINT64, FLOAT, DOUBLE, and STRING. When set to true, you can also configure u32offset_threshold (default: 0xFFFFFFFFL). If the maximum offset exceeds u32offset_threshold, an 8-byte offset file format is used; otherwise, a 4-byte format is used.

• fixed_multi_value_count: The fixed number of values for a multi-value field. When configured, the field becomes a fixed-length multi-value attribute. Supported types: int8, int16, int32, int64, uint8, uint16, uint32, uint64, float, and double. You can also use this parameter for single-value string fields, but not for multi-value string fields.

• compress_type: The compression method for storing this field as an attribute. Valid values: uniq, equal, or a combination of both. Default: empty (no compression).

• For multi-value attributes or string attributes, uniq deduplicates and compresses the data, and equal uses equal-value compression on the offset data.

• You cannot configure uniq for single-value integer attributes, but you can configure equal to apply equal-value compression to the data. Single-value FLOAT and DOUBLE attributes also support equal-value compression.

• Other compression methods: For fixed-length multi-value float fields, in addition to uniq|equal, you can also select one of the following lossy encoding compression methods: fp16, block_fp, or int8#[absMax]. For single-value float fields, you can also select fp16 or int8#[absMax] for compression.

• Both fp16 and block_fp compression provide a compression ratio of approximately 50%. The fp16 method uses slightly less space but results in a greater precision loss than the block_fp method.

• int8#[absMax]: You must specify the upper limit of the absolute value for encoding. For example, for int8#1.5, the encoded value range is [-1.5, 1.5]. This method achieves a compression ratio of 25% of the original size. The precision loss is related to absMax. A larger absMax value results in greater precision loss.

• enable_null: Specifies whether the field can contain null values. When set to true, the field cannot use fixed-length multi-value or equal compression.

• default_null_string: The literal representation of null values. Default value: __NULL__.

• For TIMESTAMP fields, configure default_time_zone in +/-HHMM format (for example, +0800 for UTC+8). When set, TIMESTAMP values without time zone information are converted to UTC based on the configured default. Summary fields display in the default time zone.

Indexes configuration

"indexs":
[
        {index1},
        {index2},
        ……
        {indexn}
]

Each item in the list is a complete index configuration. For more information about supported index types, see Inverted indexes.

Attributes configuration

"attributes": [
  "user_id", 
  "product_id", 
  "category"
]

A list of fields for forward indexes. All fields must be declared in the fields section. All field types except TEXT support attributes.

Note:

The storage formats of forward indexes for TIME, DATE, and TIMESTAMP types are as follows:

• DATE: The number of days from January 1, 1970 to the specified date, stored as a 4-byte single value. Multiply by 86,400,000 (milliseconds per day) to convert to a timestamp.

• TIME: The number of milliseconds from 00:00:00 to the specified time, stored as a 4-byte single value.

• TIMESTAMP: The number of milliseconds from January 1, 1970 to the specified timestamp, stored as an 8-byte single value.

Summaries configuration

"summarys":
{
        "summary_fields":["id", "company_id", "subject", "cat_id"],
        "compress":false 
}

• summary_fields: The fields to include in the summary. All field types are supported, and fields must be declared in the fields section.

• compress: Specifies whether to compress the summary using zlib. Default value: false.

Dictionaries configuration

"dictionaries":[
    {
        "dictionary_name":"bitmap1",
        "content":"a;an"
    },
    {
        "dictionary_name":"bitmap2",
        "content":"of;and"
    }
]

• dictionary_name: The name of the dictionary.

• content: The words in the dictionary, separated by semicolons (;).

Adaptive dictionaries configuration

"adaptive_dictionaries":[
    {
        "adaptive_dictionary_name":"df",
        "dict_type":"DOC_FREQUENCY",
        "threshold":1500000
    },
    {
        "adaptive_dictionary_name":"percent",
        "dict_type":"PERCENT",
        "threshold":30
    },
    {
        "adaptive_dictionary_name":"size",
        "dict_type":"INDEX_SIZE"
    }
]

• adaptive_dictionary_name: The name of the adaptive high-frequency dictionary rule.

• dict_type: The rule type for the adaptive high-frequency dictionary. Three types are available:

• DOC_FREQUENCY: A term is considered a high-frequency word if its document frequency (df) is greater than or equal to the threshold.

• PERCENT: A term is considered a high-frequency word if (df / totalDocCount) * 100 is greater than or equal to the threshold.

• INDEX_SIZE: A term is considered a high-frequency word if the size of the bitmap index generated for it is smaller than the size of the original index.

Note:

For inverted index fields with enumerable terms (such as a, b, and c) that are not frequently queried, set the adaptive bitmap rule to INDEX_SIZE. For frequently queried inverted indexes with non-enumerable terms, set the adaptive bitmap rule to PERCENT or DOC_FREQUENCY and select a threshold based on performance test results. As a general guideline, you can set the threshold to 5% of the total number of documents. For example, if you have 10 million documents, set the DOC_FREQUENCY threshold to 500,000 or the PERCENT threshold to 5.