Elasticsearch monitoring metrics and exception handling - Elasticsearch

Alibaba Cloud Elasticsearch provides basic monitoring metrics for running clusters, such as cluster health, query QPS, node CPU utilization, and disk usage. This topic describes how to view monitoring details and explains each metric, its common exceptions, and recommended actions.

Monitoring differences

Cluster monitoring metrics may differ from Kibana or third-party tools for the following reasons:

Sampling period differences: Cluster monitoring uses a different sampling period than Kibana and other tools, which causes data variations.
Query algorithm differences: Cluster instability affects data collection for both cluster monitoring and Kibana. For example, cluster jitter can cause the QPS metric to show spikes, negative values, or no data, while Kibana may show an empty value for the same period.

Note
If cluster monitoring provides more metrics than Kibana, use both tools for comprehensive monitoring.
Data source differences: Kibana retrieves metrics from the Elasticsearch API. Cluster monitoring collects some node-level metrics, such as CPU utilization, load_1m, and disk usage, from underlying system interfaces. These metrics reflect system-wide resource usage, not just the Elasticsearch process.

Cluster monitoring

Log on to the Alibaba Cloud Elasticsearch console.
In the left navigation menu, choose Elasticsearch Clusters.
Navigate to the target cluster.
1. In the top navigation bar, select the resource group to which the cluster belongs and the region where the cluster resides.
2. On the Elasticsearch Clusters page, find the cluster and click its ID.
In the left-side navigation pane, choose Monitoring and Logs > Cluster Monitoring.
View the monitoring details.
- View Basic Monitoring details
  
  On the Basic Monitoring tab, select a Group Name and a time range to view the corresponding monitoring details.
  Note
  
  Click Custom to view monitoring details for a custom time range.
  
  Monitoring and alerting is enabled by default. View historical data on the Cluster Monitoring page. Data has a one-minute granularity and is retained for 30 days.
  
  Basic monitoring metrics are listed in Basic monitoring metrics overview.

Basic monitoring metrics

The following tables list the basic monitoring metrics for a cluster.

Note

The actual UI may vary.

Overview

Metric name	Description
Cluster Status (value)	Indicates the health of the cluster. A value of `0.00` signifies a healthy state.
Snapshot Status (value)	The status of snapshots from the automatic backup feature. A value of `0` indicates that a snapshot exists.
Total Number of Nodes in Cluster (count)	Total number of nodes in the cluster.
Total Number of Unreachable Nodes in Cluster (count)	Total number of unreachable nodes in the cluster.
Number of Cluster Indices (count)	Number of indices in the cluster.
Number of Cluster Shards (count)	Number of shards in the cluster.
Number of Cluster Primary Shards (count)	Number of primary shards in the cluster.
Number of Slow Queries in Cluster (count)	Number of slow queries in the cluster.
Cluster Write QPS (count/s)	Number of documents written to the cluster per second.
Cluster Query QPS (count/s)	The number of queries per second (QPS) executed by the cluster. The query QPS depends on the number of primary shards in the queried indices.
Node CPU Utilization_ES Service (%)	CPU utilization of each node.
Node Heap Memory Utilization_ES Service (%)	Heap memory utilization of each node.
Node Disk Utilization (%)	Disk utilization of each node. We recommend setting the alarm threshold for disk utilization at `75%`. The utilization should not exceed `85%`.
Node Load_1m (value)	The average system load over the last `1 minute` for each node, which reflects the system's workload. A healthy value for this metric is typically less than the number of CPU cores on the node.
Node Network Bandwidth_In (KiB/s)	Rate of incoming data for each node. Monitoring period: 1 minute. Unit: KiB/s.
Node Network Bandwidth_Out (KiB/s)	Rate of outgoing data for each node. Monitoring period: 1 minute. Unit: KiB/s.
Node Network Packets_In (count)	Number of incoming network packets for each node. Monitoring period: 1 minute.
Node Network Packets_Out (count)	Number of outgoing network packets from each node. Monitoring period: 1 minute.
Node TCP Connections (count)	Number of client-initiated TCP connections to each node.
IOUtil (%)	I/O utilization for each node.
Disk Bandwidth_Read (MiB/s)	Amount of data read per second from each node in the cluster.
Disk Bandwidth_Write (MiB/s)	Amount of data written per second to each node in the cluster.
Disk IOPS_Read (count)	Number of read requests completed per second on each node in the cluster.
Disk IOPS_Write (count)	Number of write requests completed per second on each node in the cluster.

Cluster metrics

Metric name	Description
Cluster Status (value)	Indicates the health of the cluster. A value of `0.00` signifies a healthy state.
Total Number of Nodes in Cluster (count)	Total number of nodes in the cluster.
Total Number of Unreachable Nodes in Cluster (count)	Total number of unreachable nodes in the cluster.
Number of Cluster Indices (count)	Number of indices in the cluster.
Number of Cluster Shards (count)	Number of shards in the cluster.
Number of Cluster Primary Shards (count)	Number of primary shards in the cluster.
Number of Slow Queries in Cluster (count)	Number of slow queries in the cluster.
Cluster Slow Query Time Distribution	This metric is based on logs from `index.search.slowlog.query` and `index.search.slowlog.fetch`. It aggregates data by execution time (took_millis) and displays the distribution in `1-second` intervals (for example, 0–1s, 1–2s, up to 10s).
Snapshot Status (value)	The status of snapshots from the automatic backup feature. A value of `0` indicates that a snapshot exists.
Cluster Write QPS (count/s)	Number of documents written to the cluster per second.
Cluster Query QPS (count/s)	The number of queries per second (QPS) executed by the cluster. The query QPS depends on the number of primary shards in the queried indices.
Cluster Fielddata Memory Usage (B)	The amount of heap memory used by fielddata in the cluster. High usage can trigger the fielddata circuit breaker and impact cluster stability.

Index metrics

Metric name	Description
Index Bulk Write TPS (count/s)	The number of bulk requests per second for the index.
Index Query QPS (count/s)	The number of queries per second (QPS) for an index. The QPS depends on the number of primary shards in the queried index.
Index End-to-End Query Latency_max (ms)	The maximum query request time for an index, in milliseconds.

Node resource metrics

Metric name	Description
Node CPU Utilization_ES Service (%)	CPU utilization of each node. High CPU utilization, or utilization approaching 100%, can affect cluster services.
Node Heap Memory Utilization_ES Service (%)	Heap memory utilization of each node. High heap memory utilization or the presence of large memory objects can affect cluster services and automatically trigger GC operations.
Node Disk Utilization (%)	Disk utilization of each node. We recommend setting the alarm threshold for disk utilization at `75%`. The utilization should not exceed `85%`.
Node Memory Utilization_Total (%)	The system memory utilization of the node. Note This metric is supported only by the cloud-native new control plane (v3).
Node CPU I/O Wait Percentage (%)	The percentage of time the CPU spends waiting for I/O operations. Note This metric is supported only by the cloud-native new control plane (v3).
Node Load_1m (value)	The average system load over the last `1 minute` for each node, which reflects the system's workload. A healthy value for this metric is typically less than the number of CPU cores on the node.
Node CPU Utilization_Total (%)	The total CPU utilization of a node, excluding idle time. This is the sum of CPU utilization in kernel mode, user mode, and I/O wait states. Note This metric is supported only by the cloud-native new control plane (v3).

Node network metrics

Metric name	Description	Remarks
Node Network Bandwidth_In (KiB/s)	Rate of incoming data for each node in the cluster. Monitoring period: 1 minute. Unit: KiB/s.	N/A
Node Network Bandwidth_Out (KiB/s)	Rate of outgoing data for each node in the cluster. Monitoring period: 1 minute. Unit: KiB/s.	N/A
Node Network Bandwidth (KiB/s)	Node Network Bandwidth (KiB/s) = Node Network Bandwidth_In (KiB/s) + Node Network Bandwidth_Out (KiB/s).	This metric is supported only by the cloud-native new control plane (v3).
Node Network Bandwidth Utilization (%)	Node network bandwidth usage (%) = (Node network bandwidth_Input (KiB/s) + Node network bandwidth_Output (KiB/s)) / Node network base bandwidth (Gbit/s).	This metric is supported only by the cloud-native new control plane (v3).
Node TCP Connections (count)	Number of client-initiated TCP connections to each node.	N/A
Node Network Retransmission Rate (%)	The network packet retransmission rate of the node.	This metric is supported only by the cloud-native new control plane (v3).
Node Network Packets_In (count)	Number of incoming network packets for each node. Monitoring period: 1 minute.	N/A
Node Network Packets_Out (count)	Number of outgoing network packets from each node. Monitoring period: 1 minute.	N/A
Node Network Packets (count)	Node Network Packets (count) = Node Network Packets_Out (count) + Node Network Packets_In (count).	This metric is supported only by the cloud-native new control plane (v3).
Node Network Packet Utilization (%)	Node Network Packet Utilization (%) = (Node Network Packets_Out (count) + Node Network Packets_In (count)) / Node network packet transmission and reception PPS.	N/A

Node disk metrics

Metric name	Description	Remarks
Disk Bandwidth_Read (MiB/s)	Amount of data read per second from each node in the cluster.	N/A
Disk Bandwidth_Write (MiB/s)	Amount of data written per second to each node in the cluster.	N/A
Disk Bandwidth (MiB/s)	Disk Bandwidth (MiB/s) = Disk Bandwidth_Read (MiB/s) + Disk Bandwidth_Write (MiB/s).	This metric is supported only by the cloud-native new control plane (v3).
Disk Bandwidth Utilization_Cloud Disk (%)	Disk Bandwidth Utilization_Cloud Disk (%) = (Disk Bandwidth_Read (MiB/s) + Disk Bandwidth_Write (MiB/s)) / single-disk throughput of an ESSD (MiB/s).	This metric is supported only by the cloud-native new control plane (v3). For information about ESSD single-disk throughput, see ESSD.
Disk Bandwidth Utilization_Node (%)	Disk Bandwidth Utilization_Node (%) = (Disk Bandwidth_Read (MiB/s) + Disk Bandwidth_Write (MiB/s)) / Base cloud disk bandwidth of the node (Gbit/s).	This metric is supported only by the cloud-native new control plane (v3).
IOUtil (%)	I/O utilization for each node.	N/A
Disk IOPS_Read (count)	Number of read requests completed per second on each node in the cluster.	N/A
Disk IOPS_Write (count)	Number of write requests completed per second on each node in the cluster.	N/A
Disk IOPS (count)	Disk IOPS (count) = Disk IOPS_Read (count) + Disk IOPS_Write (count).	This metric is supported only by the cloud-native new control plane (v3).
Disk IOPS Utilization_Cloud Disk (%)	Disk IOPS Utilization_Cloud Disk (%) = (Disk IOPS_Read (count) + Disk IOPS_Write (count)) / Single-disk IOPS of an ESSD.	This metric is supported only by the cloud-native new control plane (v3). For information about ESSD single-disk throughput, see ESSD.
Disk IOPS Utilization_Node (%)	Disk IOPS Utilization_Node (%) = (Disk IOPS_Read (count) + Disk IOPS_Write (count)) / Base cloud disk IOPS of the node.	This metric is supported only by the cloud-native new control plane (v3).
Average request queue length	The average request queue length.	N/A

Node JVM metrics

Metric name	Description
Node Old Generation Usage (B)	Memory used by the Old Generation of the heap on each node. High usage or large objects can impact performance and trigger garbage collection (GC), potentially leading to long GC pauses or a Full GC.
Full GC Count (count)	The total number of Full GC events in the cluster within a `1-minute` period.
Node Old Generation GC Frequency (count)	The number of Old Generation GC events on each node. High usage or large objects in the Old Generation can affect cluster performance and automatically trigger GC. The collection of large objects might lead to long GC pauses or a Full GC.
Node Old Generation GC Time (ms)	The average time spent on Old Generation GC on each node. High usage or large objects in the Old Generation can automatically trigger GC. The collection of large objects might lead to long GC pauses or a Full GC.

Thread pool metrics

Metric name	Description
Search Thread Pool Active Threads (count)	The number of threads in the search thread pool that are currently executing tasks.
Search Thread Pool Rejected Requests (New) (count)	The number of rejected requests in the cluster's search thread pool.

Other metrics

Metric name	Description
Exception Count (count)	The total number of WARNING-level log entries that appear in the cluster's main log within a one-minute period.

Deprecated metrics

Metric name	Description
Search Thread Pool Rejected Requests (count)	Indicates the number of rejected requests in the query thread pool. This metric is calculated differently from the SearchThreadpoolRejectedV2 metric and is now deprecated. Use SearchThreadpoolRejectedV2 instead.

Cluster status (value)

Metric description

Indicates cluster health. A value of 0.00 means the cluster is healthy. Configure alerts for this metric. Configure cluster alerts. The following table lists the possible values.

Value	Color	Status	Description
0.00	Green	All primary and replica shards are allocated.	All indices in the cluster are healthy and have no unassigned shards.
1.00	Yellow	All primary shards are allocated, but one or more replica shards are not.	At least one index has unassigned replica shards.
2.00	Red	At least one primary shard is not allocated.	At least one index has unassigned primary shards, which means some data is unavailable.

Note

The colors in this table correspond to the cluster status displayed on the Basic Information page of your instance.

Causes for abnormal status

A value other than 0.00 indicates an abnormal cluster status. Common causes:

High CPU utilization or heap memory usage on one or more nodes, potentially reaching 100%.
High disk usage on one or more nodes, for example, exceeding 85%.
A high Load_1m on one or more nodes.
The health status of one or more indices is yellow or red.

Troubleshooting recommendations

Check the Monitoring page in the Kibana console, or view the instance logs for details. For example, delete unnecessary indices if an index consumes too much memory.
If high disk usage caused the abnormal status, see Troubleshoot and resolve high cluster disk usage and read-only issues.
For small instance types (for example, 1-core CPU and 2 GB memory), first upgrade the cluster to an instance type with a 1:4 CPU-to-memory ratio. If the status remains abnormal, follow the two recommendations above.

Snapshot status

Description

Shows the snapshot status for the automatic backup feature. A value of 0 indicates that a snapshot exists.

Value	Description
0	A snapshot exists.
-1	No snapshot exists.
1	A snapshot is in progress.
2	The snapshot task failed.

Causes of an abnormal status

A value of 2 indicates a failure. Common causes:

Disk usage on one or more nodes is high or approaching 100%.
The cluster is unhealthy.

Cluster node count

Total number of nodes in the cluster. Use this to confirm the node scale matches expectations.

Disconnected node count

Total number of disconnected nodes. Disconnected nodes can cause shard reassignments or increased query latency.

Cluster index count

Number of indices in the cluster. Too many indices can cause resource contention (memory and CPU).

Cluster shard count (count)

Number of shards in the cluster. Too many shards increase management overhead, while too few degrade query performance due to uneven load.

Cluster primary shard count

Number of primary shards in the cluster. Too few primary shards can cause a write bottleneck.

Cluster slow queries

Number of slow queries in the cluster. Use this to identify performance bottlenecks such as complex queries or index design issues.

Cluster write QPS (count/s)

Important

Sudden spikes in write QPS can cause high CPU utilization, heap memory usage, or node load, degrading cluster performance. Avoid these spikes.

Number of documents written to the cluster per second. Calculated as follows:

A single-document write request counts as 1. Multiple requests within one second are summed.
For a _bulk API request, the write QPS equals the total number of documents in the request. Multiple _bulk requests within one second are summed.

Cluster query QPS (count/s)

Important

Avoid sudden spikes in query QPS. They can cause high CPU utilization, heap memory usage, or high 1-minute load average, degrading cluster performance.

Query QPS for the cluster. The value depends on the number of primary shards in the queried index.

For example, querying an index with five primary shards counts as five separate queries.

Cluster slow query latency distribution

Metric description

Aggregates data from index.search.slowlog.query and index.search.slowlog.fetch entries. Groups queries by execution time (took_millis) in 1-second intervals (0–1s, 1–2s, up to 10s). Define your slow query threshold with the index.search.slowlog.threshold.xxx parameter. index template configuration.

Common causes for an abnormal value

If slow queries increase in specific time ranges, a service issue may exist. Common causes:

Cause	Description
High QPS	Sudden spikes or significant fluctuations in Query QPS or write QPS increase cluster load, leading to longer query execution times.
Aggregate or script queries	Aggregate queries are resource-intensive. Use them with caution.
Term queries on numeric fields	Running many term queries on numeric fields (byte, short, integer, long) can be slow because constructing the bitset for document IDs is time-consuming. If range queries or aggregations are not needed, change the field type to keyword.
Fuzzy matching	Wildcard, regex, or fuzzy queries scan the inverted index term list and collect matching document IDs, consuming substantial resources. Perform stress tests to determine an appropriate query volume.
A few individual slow query or write requests	Overall QPS fluctuations may be minor. To investigate, go to the Query Logs page and click Search Slow Log.
An excessive number of indices or shards in the cluster	Too many indices or shards can lead to high CPU utilization, HeapMemory usage, or Load_1m, degrading overall query performance.
Merge operations	Merge operations are CPU-intensive and cause a sharp drop in segment count. Monitor segment count on the Overview page for each node in the Kibana console.
Garbage collection (GC) operations	GC operations, especially full GC, free memory but consume CPU, causing spikes in CPU utilization and slower queries.
Scheduled tasks	Scheduled tasks such as data backups can consume significant I/O resources, affecting query speed.

Cluster Fielddata Memory Usage (B)

Description

Heap memory used by Fielddata in the cluster. Excessive Fielddata usage can trigger circuit breaking and affect cluster stability.

Common causes

High Fielddata usage consumes heap memory and can cause service exceptions. Common causes:

Frequent sort or aggregation operations on string (Text) fields. Fielddata for these queries is not cleared by default. Use numeric field types instead.
Sudden spikes or significant fluctuations in Query QPS or write QPS traffic. This causes Fielddata to be frequently loaded into the cache.
Too many indices or shards increase management overhead, leading to high CPU, HeapMemory, or Load_1m.

Bulk write TPS

Description

Number of bulk requests per second for an index.

Common causes for exceptions

This metric may show no data for the following reasons:

High cluster pressure interferes with monitoring data collection.
The monitoring data push failed.

IndexSearchQPS (count/s)

Description

Queries per second (QPS) for an index. The value depends on the number of primary shards in the queried index.

For example, querying an index with five primary shards counts as 5 QPS.

Causes of abnormal values

This metric may show no data. Common reasons include:

High cluster load can interfere with monitoring data collection.
The monitoring data push failed.

Important

A sudden spike in IndexSearchQPS can indicate that an index is causing high CPU, heap memory, or Load_1m, affecting cluster stability. Consider optimizing the index.

IndexSearchDelayMax (ms)

Maximum query delay on an index, in milliseconds.

Node CPU utilization (%)

Metric description

CPU utilization for each node. High utilization, especially near 100%, can impact cluster services.

Common causes of exceptions

A spike or significant fluctuation indicates a service problem. Common causes:

Cause	Description
QPS	Sudden spikes or large fluctuations in Query QPS or write QPS traffic.
Slow query or write requests	QPS fluctuations may be minor. To investigate, go to the LogSearch page and click Search Slow Log.
Excessive indices or shards	Too many indices or shards increase management overhead, leading to high CPU, HeapMemory, or Load_1m.
Cluster merge operations	Merge operations consume CPU and cause a sharp drop in segment count. Check segment count on the node's Overview page in the Kibana console.
GC operations	GC operations, especially full GC, free memory but are CPU-intensive, causing spikes in CPU utilization.
Scheduled tasks	Running scheduled tasks, such as a data backup or other custom jobs, can be resource-intensive.

Note

Node CPU utilization includes resource consumption from both system-level processes and Elasticsearch tasks.

Node disk usage (%)

Disk usage for each node. Keep disk usage below 75% and do not exceed 85%. Exceeding these thresholds can affect cluster services.

Disk usage	Description
>85%	The cluster prevents new shards from being allocated to the node.
>90%	The cluster attempts to relocate shards from the node to other data nodes with lower disk usage.
>95%	Elasticsearch applies the `read_only_allow_delete` setting to all indices on the node. This action blocks write operations. You can only read data from or delete the affected indices.

Important

Configure monitoring alerts for this metric. If triggered, promptly scale up disks and nodes or clear index data to prevent disruptions.
Node disk usage includes resources used by both system-level processes and Elasticsearch tasks.

Node heap memory usage (ES service) (%)

Description

Heap memory usage for each node. High usage or large memory objects can affect performance and trigger GC operations.

Causes of abnormal values

A sudden spike or significant fluctuation often indicates a service anomaly. Common causes:

Cause	Description
QPS	Sudden spikes or large fluctuations in Query QPS or write QPS.
A few slow query requests	QPS fluctuations may be minor. To investigate, analyze the Search Slow Log on the LogSearch page.
A large number of slow write requests	QPS shows significant fluctuations. To investigate, analyze the Indexing Slow Log on the LogSearch page.
The cluster has too many indices or a high total number of shards	Too many indices or shards increase management overhead, leading to high CPU, heap memory, or Load_1m.
Merge operation	Merge operations are CPU-intensive and cause a sharp drop in segment count. Check the Overview page for the node in the Kibana console.
GC operation	A GC operation, such as a Full GC, frees up memory but consumes CPU resources. This can cause a sharp drop in heap memory usage.
Scheduled task	For example, a data backup or other custom task.

Node Load_1m

Description

1-minute load average for each node, indicating system workload. A normal value is less than the number of CPU cores. The following table explains values for a single-core node.

Node Load_1m	Description
<1	No processes are waiting for resources.
=1	The system is fully utilized, with no capacity for additional processes.
>1	Processes are queued, waiting for resources.

Note

The Node Workload Within One Minute metric includes resource consumption from both system-level processes and Elasticsearch tasks.
Fluctuations in the Node Workload Within One Minute metric are expected. For a more accurate analysis, focus on the Node CPU usage metric.

Abnormal causes

A value exceeding the number of CPU cores indicates system overload. Common causes:

CPU utilization or heap memory usage on the node is excessively high, potentially reaching 100%.
A sudden spike or significant increase in Query QPS or write QPS.
Expensive slow queries.

Use the Log Query page to analyze these logs.

Note

The Node Load_1m metric includes resource consumption from both system-level processes and Elasticsearch tasks.

Node memory usage_total (%)

System memory usage for the node.

Node CPU IO wait percentage (%)

Percentage of time the node's CPU spends waiting for I/O operations.

Node inbound packets (count)

Number of inbound network packets for each node. Monitoring cycle: 1 minute.

Node network packets out (count)

Number of packets sent from each node per minute.

Node inbound bandwidth (KiB/s)

Inbound data rate for each node. Monitoring cycle: 1 minute. Unit: KiB/s.

Node network bandwidth_output (KiB/s)

Outbound network bandwidth for each node, in KiB/s. Updated every minute.

Node TCP connections

Description

Number of established TCP connections from clients to each node.

Abnormal causes

A spike often occurs when clients fail to release TCP connections promptly. Configure client-side policies to release idle connections.

IOUtil (%)

Description

I/O utilization for each node.

Abnormal causes

High disk usage increases read/write wait times and can cause I/O utilization spikes up to 100%. Analyze your workload and consider upgrading the cluster configuration.

Node network retransmission rate (%)

Percentage of network packets retransmitted by the node.

Node network bandwidth (KiB/s)

Sum of node network bandwidth_Input and node network bandwidth_Output.

Node network bandwidth usage (%)

Node network bandwidth usage (%) = (Node Network Bandwidth_Input (KiB/s) + Node Network Bandwidth_Output (KiB/s)) / Node network base bandwidth (KiB/s).

Node network packets (count)

Sum of node network packet_output and node network packet_input.

Node network packet usage (%)

Node network packet usage (%) = (node network packet_outbound (PPS) + node network packet_inbound (PPS)) / maximum network packets per second (PPS).

Disk bandwidth read (MiB/s)

Data read from each node per second.

Disk bandwidth_write (MiB/s)

Write bandwidth for each node.

Disk read IOPS

Read requests completed per second on each node.

Disk IOPS write

Write requests completed per second by each node.

Average request queue length

The average length of the request queue.

Disk bandwidth (MiB/s)

Disk bandwidth (MiB/s) = disk bandwidth_read (MiB/s) + disk bandwidth_write (MiB/s).

Cloud disk bandwidth usage (%)

Disk Bandwidth Usage_Cloud Disk (%) = (Disk Bandwidth_Read (MB/s) + Disk Bandwidth_Write (MB/s)) / Single Disk Throughput (MB/s).

Disk bandwidth usage_node (%)

Calculated as (disk bandwidth_read (MiB/s) + disk bandwidth_write (MiB/s)) / disk basic bandwidth (Gbit/s). Convert all values to the same units.

Disk IOPS (count)

Disk IOPS (count) = disk read IOPS (count) + disk write IOPS (count).

Disk IOPS usage (Cloud) (%)

Disk IOPS usage_disk (%) = (Disk IOPS_Read (count) + Disk IOPS_Write (count)) / single disk IOPS capacity.

Disk IOPS usage_node (%)

disk IOPS usage_node (%) = (disk IOPS_read (count) + disk IOPS_write (count)) / cloud disk basic IOPS

Node old generation usage (B)

Metric description

Old generation heap memory used by each node. High usage or large objects can affect performance and trigger GC, potentially leading to long pauses or full GC.

Causes of metric anomalies

A sudden spike or significant fluctuation often indicates a service exception. Common causes:

Cause	Description
QPS	Sudden spikes or significant fluctuations in Query QPS or write QPS.
Aggregation or script queries	Aggregation queries are resource-intensive. Use them with caution.
Term queries on numeric fields	Running many term queries on numeric fields (byte, short, integer, long) can be slow due to time-consuming bitset construction. If range queries or aggregations are not needed, change the field type to keyword.
Fuzzy matching	Wildcard, regex, or fuzzy queries scan the inverted index term list and collect matching document IDs, consuming substantial resources. Perform stress tests to determine an appropriate query volume.
A few slow queries	QPS fluctuations may be minor. To investigate, go to the query log page and click Search Slow Log.
A few slow write requests	QPS fluctuations may be minor. To investigate, go to the query log page and click Indexing Slow Log.
An excessive number of indices or shards in the cluster	Too many indices or shards can cause high CPU, heap memory, or Load_1m.
Merge operations	Merge operations are CPU-intensive and cause a sharp drop in segment count. Monitor this on the Overview page for each node in the Kibana console.
GC operations	A GC operation, such as a full GC, frees up memory but consumes CPU resources. This can cause a sudden drop in heap memory usage.
Scheduled tasks	Data backup or other custom tasks.

Full GC count

Important

Frequent Full GC events can degrade cluster performance.

Description

Number of full GC events in the cluster per minute.

Causes for abnormality

A value greater than zero indicates a service exception. Common causes:

High heap memory usage.
Large memory objects.

Node Old GC count

Description

Counts old generation GC events on each node. High usage or large objects can trigger automatic GC, potentially causing long pauses or full GC.

Note

The Full GC metric for basic monitoring is sourced from logs, while memory metrics in advanced monitoring are collected by the ES engine. To account for these different data sources, evaluate cluster performance by combining all available metrics.

Common causes

See Node Old Area Usage (B).

Node Old GC duration (ms)

Metric

Average duration of old generation GC on each node. High old generation usage or large objects can trigger GC, causing longer durations or full GC.

Causes for abnormal values

See JVMMemoryOldUsedBytes.

Search threadpool active threads (count)

Active threads in the cluster's query thread pool.

Rejected requests in query thread pool (count)

This deprecated metric counts rejected requests in the cluster's query thread pool. Use SearchThreadpoolRejectedV2 instead.

Rejected query requests

Rejected requests in the cluster's query thread pool. When the thread pool is full, new query requests are rejected.

Exception count

Metric description

Total warning-level log entries in the cluster logs within a one-minute period.

Causes for abnormal values

A value other than 0 indicates a service exception. Common causes:

Abnormal query requests.
Abnormal write requests.
Errors in Elasticsearch tasks.
Garbage collection operations.

Troubleshooting

Go to the Query logs page and click Cluster Log. On the Cluster Log page, review the exception details to find the root cause.

Note

If the Cluster Log contain GC records, these records are also included in the NodeStatsExceptionLogCount metric.