The Lindorm search engine monitors index status, query latency, and resource levels in real time, sending proactive alerts to keep your search service stable and efficient.
Search engine monitoring metrics
Total CPU utilization
This metric is the sum of CPU user (process usage), CPU system (system usage), and CPU wio (I/O wait). If the total utilization remains above 90% for an extended period, read and write requests to the search engine will queue up, increasing response time (RT) and limiting throughput. Under a CPU bottleneck, server-side performance does not improve even if you increase client concurrency.
High CPU utilization is typically caused by an increase in read/write traffic or large queries (often seen as a sharp rise in CPU user), or by I/O-intensive workloads such as tokenization queries (often seen as a significant rise in CPU wio). To troubleshoot, check the following:
-
Check for an increase in read/write traffic. Focus on the node-level, server-side metrics for
write TPSandquery QPS. -
Check for changes in query patterns that might be causing large queries. Focus on the node-level, server-side metrics for Query phase latency and Fetch phase latency.
For latency-insensitive workloads, prioritize monitoring the search thread rejected and write thread rejected metrics. These metrics directly reflect rejections caused by a full request queue, which clients experience as failed read and write requests. Configure alerts for sustained non-zero values rather than brief spikes.
Memory backpressure
Before executing a request, the Lindorm search engine server checks if sufficient memory is available based on current JVM memory usage and the request size. If the free memory is insufficient and a Full GC cannot free up enough space, the request is rejected. The Total Memory Backpressure Count metric on the node-level service dashboard directly reflects the frequency of these rejections. Similar to read/write thread rejections, if client traffic drops or large queries stop, the system gradually restores the JVM memory level through garbage collection, and the backpressure frequency decreases to zero. Therefore, we recommend configuring an alert for when this metric remains in a non-zero state for an extended period.
Service query and write latency
This metric, available as search latency or bulk write latency under the node-level, business-side metrics, directly reflects the request response time (RT), including the mean and p99, when a client calls the search/bulk API. For latency-sensitive workloads, use an alert template to configure alerts for the mean or p99 latency of search/bulk requests.
Create an alert rule
Log on to the Lindorm console. In the upper-left corner of the page, select the region of the instance. On the Instances page, click the ID of the target instance or click View Instance Details in the Actions column for the instance.
-
In the left-side navigation pane, click Alerts. The Lindorm alert list page opens, showing the alert rules for the instance.
-
Click Create Lindorm alert rule.
-
On the Create Lindorm alert rule page, configure the following parameters.
Parameter
Description
Alert Name
The name of the alert rule.
Check Type
Select Static threshold.
Lindorm cluster
The current instance to which this alert rule applies.
Alert Contact Group
Select
Lindorm-Search 3.0.Alert metric
Multiple metrics are available, including Total CPU utilization, Memory backpressure, and Service query and write latency.
In the Alert condition section, set the threshold, for example, greater than 0.