OpenSearch:Configure online services for an index table

Configuration example

{
  "realtime": true,
  "cluster_config": {
    "table_name": "my_table"
  },
  "online_index_config": {
    "online_keep_version_count": 2,
		"on_disk_flush_realtime_index": true,
		"enable_async_dump_segment": true,
    "build_config": {
      "build_total_memory": 2048
    },
    "max_realtime_memory_use": 4096,
  	"need_read_remote_index": true,
  	"need_deploy_index": true,
    "load_config": [
      {
        "file_patterns": [
          "/index/title",
          "/attribute/.*",
        ],
        "load_strategy": "mmap",
        "load_strategy_param": {
          "slice": 409600,
          "lock": true,
          "interval": 2
        },
        "remote": false,
        "deploy": true
      }
    ],
    "speedup_primary_key_reader": true
  },
  "build_option_config": {
    "async_queue_size": 10000,
    "max_recover_time": 600,
    "async_build": true
  }
}

Parameter descriptions

• realtime: Specifies whether to enable real-time mode. Valid values: true and false.

• cluster_config:

  • table_name: The name of the index table.

• online_index_config: Online control parameters for the index table, including real-time build parameters and index loading policies.

  • online_keep_version_count: The number of incremental index versions to retain for online services. The system deletes the oldest versions when the count exceeds this value.

  • on_disk_flush_realtime_index: Specifies whether to flush the real-time index to disk. Valid values: true and false. Default value: false. When enabled, the system dumps the index to the local disk when the real-time index size reaches the build_total_memory value, and then loads it based on the load_config policy. This allows the same amount of real-time memory to support more real-time indexes. Recommended for applications with large data updates.

  • enable_async_dump_segment: Specifies whether to enable asynchronous dumping for real-time indexes. Valid values: true and false. Default value: false. When enabled, dumping does not block the real-time index build. Recommended value: true.

  • build_config: Build parameters for the real-time index.

    • build_total_memory: The total memory for the real-time index build, in MB. When this limit is exceeded, a real-time index dump is triggered. The on_disk_flush_realtime_index parameter determines whether the index is dumped to memory or disk. Note: This value must be less than the max_realtime_memory_use value.

  • max_realtime_memory_use: The total memory for the real-time index, in MB, including memory for the index being built and for already-dumped indexes. The build process stops when memory usage reaches this limit. After an incremental index merge switches real-time data to online services, the system clears the data from the real-time index and releases the memory. Note: This value must be greater than the build_total_memory value.

  • need_read_remote_index: Specifies whether to enable storage-compute separation. Default value: false. When enabled, data nodes read index data directly from distributed storage, which reduces startup time, increases data capacity, and saves costs, but affects query performance.

  • need_deploy_index: Specifies whether to deploy the index to the local disk. Default value: true. Deploying the index to the local disk in advance can improve retrieval performance.

  • load_config: The loading policy for the index table, which controls how the index is loaded into memory for fine-grained memory control, reduced memory overhead, and improved retrieval performance. For more information, see Index table loading policies. The need_read_remote_index, need_deploy_index, and load_config parameters work together to minimize the performance impact of storage-compute separation while saving costs.

• build_option_config: Process-level parameters for online services to load the index table.

  • async_build: Specifies whether to use asynchronous mode for reading real-time data and for the build process. Default value: false. Recommended value: true for applications sensitive to timeliness or with a high volume of data updates.

  • async_queue_size: The work queue size when async_build is set to true. Default value: 1000.

  • max_recover_time: The maximum time in seconds for a data node to catch up on real-time data from the message queue after the process starts. For example, a value of 600 means the process spends up to 10 minutes catching up on data after loading the index. After this period, the service becomes available even if a data delay remains. If there is little real-time data and no delay is detected, the process immediately sets its status to Normal and begins serving requests.