An index table schema defines how OpenSearch Retrieval Engine Edition stores and indexes your documents. In the schema, you specify field definitions, inverted indexes, forward indexes (attributes), summary indexes, dictionaries, and time to live (TTL) settings. The system generates a configuration file named $table_name_schema.json for each index table.
Schema structure
The top-level structure of an index table schema:
{
"table_name": "sample", // Index table name — also used in the generated schema filename
"fields": [], // Field definitions (field name, type, analyzer)
"indexs": [], // Inverted index configurations
"attributes": [], // Forward index (attribute) configurations
"summarys": {}, // Summary index configuration
"dictionaries": [], // Dictionaries for bitmap indexes
"adaptive_dictionaries": [], // Rules for generating adaptive bitmap indexes
"enable_ttl": true, // Whether to enable TTL; default: false
"ttl_field_name": "ttl_field", // Field that holds per-document TTL values
"default_ttl": 86400 // Default TTL in seconds
}Top-level parameters
| Parameter | Required | Description | Default |
|---|---|---|---|
table_name | Yes | Name of the index table. Used to identify the table; the system generates $table_name_schema.json. | — |
fields | Yes | Field definitions — name, type, analyzer, and storage options. | — |
indexs | Yes | Inverted index configurations. | — |
attributes | No | Forward index (attribute) field list. Fields of all types except TEXT can be attributes. | — |
summarys | No | Summary index configuration — which fields to store for retrieval. | — |
dictionaries | No | Static dictionaries for bitmap indexes. Omit if no bitmap index is needed. | — |
adaptive_dictionaries | No | Rules for auto-generating adaptive high-frequency bitmap indexes. | — |
file_compress | No | Compression method declarations and aliases for forward, inverted, and summary index files. | — |
enable_ttl | No | Enable TTL. Expired documents are automatically removed. | false |
ttl_field_name | No | Field that holds per-document TTL values (UINT32, single-value). If blank in a document, default_ttl applies. If not set, the built-in field ops_doc_time_to_live_in_seconds is used. | ops_doc_time_to_live_in_seconds |
default_ttl | No | Default TTL in seconds. If enable_ttl is set but default_ttl is not, the value std::numeric_limits<int64_t>::max() >> 20 is used. Setting default_ttl without enable_ttl automatically sets enable_ttl to true. | std::numeric_limits<int64_t>::max() >> 20 |
fields configuration
The fields array declares every field available for indexing, attribute storage, or summary retrieval.
"fields": [
{
"field_name": "title",
"field_type": "TEXT",
"analyzer": "chn_standard" // Required for TEXT fields; not allowed for other types
},
{
"field_name": "dup_title",
"field_type": "TEXT",
"analyzer": "fuzzy",
"user_defined_param": {
"copy_from": "title" // Copies data from the "title" field
}
},
{
"field_name": "category",
"field_type": "INTEGER",
"multi_value": true,
"compress_type": "uniq|equal" // Remove duplicates + compress offset data
},
{
"field_name": "mlr_features",
"field_type": "INTEGER",
"multi_value": true,
"updatable_multi_value": true // Allows in-place updates to multi-value data
},
{
"field_name": "feature",
"field_type": "float",
"multi_value": true,
"fixed_multi_value_count": 32, // Fixed-length multi-value: exactly 32 values per document
"compress_type": "uniq|fp16", // fp16: ~50% compression, accepts slight precision loss
"updatable_multi_value": true
},
{
"field_name": "price",
"field_type": "INTEGER",
"enable_null": true,
"default_null_string": "default_null" // String representing a null value in the source document
},
{
"field_name": "timestamp",
"field_type": "TIMESTAMP",
"default_time_zone": "+0800" // UTC+8; applied when source document omits timezone
}
]Field parameters
| Parameter | Required | Description | Default |
|---|---|---|---|
field_name | Yes | Field name. | — |
field_type | Yes | Field type. See Built-in field types in OpenSearch Retrieval Engine Edition. | — |
analyzer | Conditional | Analyzer for TEXT fields. Required for TEXT; not allowed for other types. See Analyzers. To apply a different analyzer to a TEXT field, create a derived field and set copy_from in user_defined_param. | — |
multi_value | No | Marks the field as multi-value. When configured as an attribute, a multi-value attribute is created. | false |
updatable_multi_value | No | Allows in-place updates to multi-value field data. Also applies to single-value STRING fields. Supported types: INT8, UINT8, INT16, UINT16, INTEGER, UINT32, LONG, UINT64, FLOAT, DOUBLE, STRING. | false |
fixed_multi_value_count | No | Fixed number of values per document. Supported types: INT8, INT16, INT32, INT64, UINT8, UINT16, UINT32, UINT64, FLOAT, DOUBLE. Cannot be set for multi-value STRING fields. | — |
compress_type | No | Compression method when the field is stored as an attribute. See Compression methods below. | "" (no compression) |
enable_null | No | Allow null values. When true, the field cannot be fixed-length multi-value or single-value, and compress_type cannot be equal. | false |
default_null_string | No | String literal representing a null value in the source document. | "__NULL__" |
default_time_zone | No | For TIMESTAMP fields only. Format: +/-HHMM. Values without timezone information are interpreted in this timezone. If the field is stored as a summary field, display follows this timezone. | — |
Compression methods
Use compress_type to reduce attribute storage size. The trade-offs depend on your field type and acceptable precision loss.
compress_type value | Applicable field types | Behavior |
|---|---|---|
uniq | Multi-value attributes, STRING attributes | Removes duplicate values before storing. |
equal | All single-value attributes (INTEGER, FLOAT, DOUBLE, etc.); offset data for multi-value attributes | Equal-value compression on offset data. Not supported for single-value INTEGER using uniq. |
uniq|equal | Multi-value attributes | Applies both: deduplication + offset compression. |
fp16 | Fixed-length multi-value FLOAT; single-value FLOAT | ~50% compression. Slightly smaller encoding space than block_fp, but greater precision loss. |
block_fp | Fixed-length multi-value FLOAT | ~50% compression. Slightly larger encoding space than fp16, but lower precision loss. |
int8#[absMax] | Fixed-length multi-value FLOAT; single-value FLOAT | ~75% compression (value compressed to 25% of original size). Precision loss increases with larger absMax. Example: int8#1.5 encodes the range [−1.5, 1.5]. |
Whenupdatable_multi_valueistrue, the field uses a 4-byte offset format by default (u32offset_threshold: 0xFFFFFFFFL). If the maximum offset exceeds this threshold, the format switches to 8 bytes. Most configurations do not require changing this threshold.
indexs configuration
The indexs array holds a list of inverted index configurations. Each element is a complete index definition:
"indexs": [
{ /* index 1 configuration */ },
{ /* index 2 configuration */ }
]For supported index types and configuration details, see Overview of inverted indexes.
attributes configuration
The attributes array lists the field names to store as forward indexes. All field types except TEXT can be used as attributes. Fields must be declared in fields.
"attributes": [
"user_id",
"product_id",
"category"
]DATE, TIME, and TIMESTAMP storage format
Forward index attributes of these types store numeric values, not formatted strings:
| Field type | Stored as | Storage size | Conversion |
|---|---|---|---|
| DATE | Days since January 1, 1970 | 4 bytes | Multiply by 86,400,000 to get a millisecond timestamp. |
| TIME | Milliseconds since 00:00:00 | 4 bytes | — |
| TIMESTAMP | Milliseconds since January 1, 1970 | 8 bytes | — |
summarys configuration
The summarys object specifies which fields to store verbatim for retrieval at query time. All field types can be included. Fields must be declared in fields.
"summarys": {
"summary_fields": ["id", "company_id", "subject", "cat_id"],
"compress": false
}| Parameter | Required | Description | Default |
|---|---|---|---|
summary_fields | Yes | Fields to include in the summary index. | — |
compress | No | Compress the summary using zlib. true enables compression; false disables it. | false |
dictionaries configuration
The dictionaries array defines static high-frequency term dictionaries for bitmap indexes. Bitmap indexes reduce index space and improve retrieval performance for high-frequency terms. Omit this section if no bitmap index is needed.
"dictionaries": [
{
"dictionary_name": "bitmap1",
"content": "a;an" // Terms separated by semicolons
},
{
"dictionary_name": "bitmap2",
"content": "of;and"
}
]| Parameter | Required | Description |
|---|---|---|
dictionary_name | Yes | Name of the dictionary. |
content | Yes | All terms in the dictionary, separated by semicolons (;). |
adaptive_dictionaries configuration
The adaptive_dictionaries array defines rules for automatically generating high-frequency term bitmap indexes based on document statistics. Use this when you want the system to determine which terms qualify as high-frequency rather than specifying them manually.
"adaptive_dictionaries": [
{
"adaptive_dictionary_name": "df",
"dict_type": "DOC_FREQUENCY",
"threshold": 1500000 // Terms with df >= 1,500,000 become high-frequency
},
{
"adaptive_dictionary_name": "percent",
"dict_type": "PERCENT",
"threshold": 30 // Terms where (df / totalDocCount) * 100 >= 30 become high-frequency
},
{
"adaptive_dictionary_name": "size",
"dict_type": "INDEX_SIZE" // Compares bitmap index size vs original; uses smaller option
}
]| Parameter | Required | Description |
|---|---|---|
adaptive_dictionary_name | Yes | Name of the adaptive dictionary rule. |
dict_type | Yes | Rule type for identifying high-frequency terms. See the table below. |
threshold | Conditional | Required for DOC_FREQUENCY and PERCENT. Not used by INDEX_SIZE. |
Choosing a dict_type
dict_type | How high-frequency terms are identified | When to use |
|---|---|---|
DOC_FREQUENCY | Terms with document frequency (df) >= threshold | Non-enumerable terms queried frequently (e.g., common stop words at scale) |
PERCENT | Terms where (df / totalDocCount) x 100 >= threshold | Non-enumerable terms queried frequently; relative threshold adjusts automatically as index grows |
INDEX_SIZE | Terms where the bitmap index is smaller than the original inverted index | Enumerable terms (e.g., category codes, status flags) that are infrequently used in queries |
Setting the threshold: Base your threshold on performance testing. As a starting point, target the top 5% of your document corpus. For a 10-million-document index: set DOC_FREQUENCY threshold to 500000 and PERCENT threshold to 5.