SysOM (System Observer Monitoring) provides Container Monitoring observability at the operating system kernel layer and supports containerized deployment, migration, and monitoring. This topic describes how to enable and use the ack-sysom-monitor feature and explains the related monitoring metrics.
Prerequisites
-
You have created an ACK managed cluster or an ACK serverless cluster that was created after October 2021, and you have made sure that the cluster version is 1.18.8 or later. If you need to upgrade a cluster, see Manually upgrade a cluster.
-
You have enabled Managed Service for Prometheus.
Overview of the ack-sysom-monitor add-on
ack-sysom-monitor is the monitoring add-on of SysOM. Based on eBPF technology, it collects kernel-level metrics at both the node and container dimensions. In addition to basic system metrics, it provides in-depth enhanced metrics that detect typical issues such as system jitter, latency, resource leaks, and abnormal Pod memory.
Billing of the ack-sysom-monitor monitoring feature
After you enable the ack-sysom-monitor monitoring feature, the related components automatically send monitoring metrics to Managed Service for Prometheus. These metrics are treated as custom metrics. Using custom metrics incurs additional fees.
To avoid additional fees, before you enable this feature, carefully read Billing overview of Managed Service for Prometheus to understand the billing policy of custom metrics. Fees vary based on factors such as your cluster size and the number of applications. You can use the resource consumption statistics feature to monitor and manage your resource usage.
Enable the ack-sysom-monitor monitoring feature
-
Log on to the ARMS console.
-
In the left-side navigation pane, click Integration Center.
-
On the Integration Center page, in the Infrastructure section, click SysOM.
-
On the SysOM panel, on the Start Access tab, select the container cluster that you want to connect, and then click OK.
Use the ack-sysom-monitor monitoring feature
Procedure
-
Log on to the ACK console. In the left navigation pane, click Clusters.
-
On the Clusters page, click the name of your cluster. In the left navigation pane, click .
-
On the Prometheus Monitoring page, click the SysOM tab to view the ack-sysom-monitor monitoring feature.
ack-sysom-monitor provides operating system kernel-level monitoring at the node and Pod dimensions.
-
Node-dimension operating system kernel-level monitoring
On the SysOM - Nodes tab, you can monitor the metrics of modules such as memory, CPU and scheduling, storage, and network from the node dimension.
-
Pod-dimension operating system kernel-level monitoring
On the SysOM - Pods tab, you can monitor the metrics of modules such as memory, CPU, network, and I/O in real time from the Pod dimension.
-
Related operations
To disable SysOM kernel-level Container Monitoring, you can uninstall the ack-sysom-monitor add-on to avoid additional billing. For the procedure, see Components.
Metric descriptions
The monitoring metrics of ack-sysom-monitor are defined in the standard Prometheus metric format.
Node metrics
Node metrics are categorized into CPU, memory, storage, network, and other metrics.
CPU and scheduling metrics
|
Metric name |
Metric type |
Unit |
Description |
|
sysom_proc_cpu_total |
gauge |
% |
Shows the proportion of overall CPU running time on the node. This metric indicates the ratio of the running time of all CPUs in each state to the total CPU running time. These states include user mode, kernel mode, software interrupt (softirq), hardware interrupt (hardirq), idle, and I/O wait (iowait). |
|
sysom_proc_cpus |
gauge |
% |
Shows the proportion of running time of a specific CPU on the node. This metric indicates the ratio of the running time of a specific CPU in different system states to the total CPU running time. These states include user mode, kernel mode, software interrupt (softirq), hardware interrupt (hardirq), idle, and I/O wait (iowait). |
|
sysom_proc_sirq |
gauge |
% |
Shows the software interrupt status of the node. This metric indicates the number of times various types of software interrupts (softirq) occur in the system. The counted software interrupt types include HI (high priority), TIMER (timer), NET_TX (network transmit), NET_RX (network receive), BLOCK (block device), IRQ_POLL (interrupt polling), TASKLET (task queue), SCHED (scheduler), HRTIMER (high-resolution timer), and RCU software interrupts. |
|
sysom_proc_stat_counters |
gauge |
- |
Shows whether the node has too many processes in the D state, to help you understand the system load. This metric indicates the number of processes in different states (Running and D state) in the system. It also provides the system startup time and the number of context switches. |
|
sysom_proc_loadavg |
gauge |
- |
The average load of the node. This metric indicates the average load (load average) of the system, including the length of the run queue (runq), the average load over the last 1 minute, the average load over the last 5 minutes, the average load over the last 15 minutes, and the total number of processes in the system. |
|
sysom_proc_schedstat |
gauge |
ns (nanoseconds) |
Shows the scheduling latency of the node. This metric provides statistics related to CPU scheduling, including the time consumed by processes waiting to be scheduled in the scheduling queue of the current CPU, and the length of the time slice running on the current CPU. |
|
sysom_cpu_dist |
gauge |
- |
Shows the overall scheduling status of the node. This metric provides statistics on the time elapsed from when a process yields the CPU to when it is next scheduled to run on the CPU. It includes the number of times all processes in the system went from yielding the CPU to being scheduled to run on the CPU again within different time intervals (1us, 10us, 100us, 1ms, 10ms, 100ms, and 1s). |
Memory metrics
|
Metric name |
Metric type |
Unit |
Description |
|
sysom_proc_meminfo |
gauge |
KiB |
Shows the usage of various types of memory on the node. This metric shows memory usage, including but not limited to total memory (Total), free memory (Free), available memory (Available), page cache (Cache), buffer cache (Buffers), reclaimable memory (SReclaimable), and unreclaimable memory (SUnreclaim). |
|
sysom_proc_vmstat |
gauge |
- |
Shows the detailed memory usage and memory events of the node. This metric indicates page-based system memory statistics and statistics of some memory events. The memory information and event statistics include: the number of free pages (Free Pages), the number of dirty pages (Dirty Pages), the number of pages read from or written to the disk (Pages Read/Write), the number of pages reclaimed from the inactive list (Pages Reclaimed from Inactive List), and the number of out-of-memory (OOM) kills. |
|
sysom_proc_buddyinfo |
gauge |
- |
Shows statistics on the allocation and release of the buddy system in the kernel memory management of the node. This metric provides details about the kernel buddy system, including all memory nodes (Node) and zones (Zone) in the system, and the number of page blocks in the buddy system lists of different page block sizes. |
Storage metrics
|
Metric name |
Metric type |
Unit |
Description |
|
sysom_proc_disks |
gauge |
- |
Shows statistics such as reads, writes, I/O requests, and latency of each disk and partition on the node, to help you understand the I/O status of the node. This metric provides statistics about disks and partitions in the system, specifically including the number of completed read and write requests of disk partitions, the total time spent on read and write requests, the number of merged read and write requests, and the number of incomplete (Inflight) I/O operations on the device. |
|
sysom_fs_stat |
gauge |
- |
Shows the usage of mounted file systems on the node. This metric provides usage statistics of each file system (FS) in the system, specifically including the mount target, block size, number of used and available blocks, and number of used and available inodes of the file system. |
Network metrics
|
Metric name |
Metric type |
Unit |
Description |
|
sysom_proc_networks |
gauge |
- |
Shows the traffic of each network interface card (NIC) on the node. This metric provides traffic statistics of each NIC in the system, specifically including the total number of packets sent/received by the NIC, the total number of bytes, the total number of packets dropped by the device driver during sending/receiving, and the total number of errors during packet sending/receiving. |
|
sysom_proc_pkt_status |
gauge |
- |
Shows the overall status of how the network protocol stack of the node processes data packets. This metric provides the number of events that occur when network packets pass through the network protocol stack in the system, including the number of packet drops in the network protocol stack, the number of stack overflows, and the number of assertion failures. |
|
sysom_sock_stat |
gauge |
- |
This metric helps detect situations such as socket and buffer exhaustion caused by issues in application logic or system parameter settings. This metric provides statistics on the usage of sockets and their buffers on the node, including the socket usage of total/raw/tcp/udp, the number of sockets in the TCP time wait or orphan state, and the memory usage of TCP/UDP sockets. |
|
sysom_softnets |
gauge |
- |
Shows the status of packets received by the NIC software interrupt of each CPU on the node. This metric provides statistics related to network software interrupts in the system, including the number of packets received and dropped in software interrupts on a CPU, and the number of calls to the network packet-receiving software interrupt handler net_rx_action. |
|
sysom_net_health_hist |
gauge |
- |
Shows the round-trip time distribution of all TCP connections on the node, to help you understand the latency of TCP links. This metric provides the round-trip time distribution of all TCP connections on the node. Specifically, it includes statistics on the number of times the smoothed RTT (Round Trip Time) falls within different time ranges, such as within 10ms, 100ms, and 1s. |
|
sysom_net_health_count |
gauge |
- |
Similar to the |
|
sysom_net_retrans_count |
gauge |
- |
Shows the retransmission status of all TCP connections on the node, and helps you quickly understand the causes of retransmissions. This metric provides the types of packets retransmitted by TCP connections in the system and the corresponding counts, including the number of packets retransmitted due to RTO (Retransmission Timeout) and the number of retransmissions of different packet types (such as Syn, Syn-ack, and Reset). |
|
sysom_net_tcp_count |
gauge |
- |
Shows the basic status of TCP connections on the node. This metric provides statistics related to TCP connections in the system, specifically including the number of established and active TCP connections in the system, the number of TCP segments received/sent, the number of retransmitted TCP segments, and the number of packets with errors on the receiver side. |
|
sysom_net_udp_count |
gauge |
- |
Shows the basic status of UDP connections on the node. This metric provides statistics related to UDP connections in the system, including the number of UDP packets sent/received in the system, the number of errors in the UDP send/receive buffers, and the number of error packets caused by no available ports. |
|
sysom_net_ip_count |
gauge |
- |
Shows the basic status of the IP layer on the node. This metric provides statistics related to the IP layer in the system, including the number of IP packets forwarded, received, and sent in the system. |
|
sysom_net_icmp_count |
gauge |
- |
Shows the basic status of the ICMP protocol on the node. This metric provides statistics related to the ICMP protocol in the system, including the number of ICMP packets sent/received in the system and the number of packets with send/receive errors. |
Other system metrics
|
Metric name |
Metric type |
Unit |
Description |
|
sysom_cgroups |
gauge |
- |
Shows the number of cgroups used by each cgroup subsystem in the system, to help you observe whether there is a cgroup leak. This metric provides the number of cgroups under each cgroup subsystem in the system, including the number of cgroups in the directories of cgroup subsystems such as CPU, Cpuacct, Memory, Pids, Blkio, and Devices. |
|
sysom_uptime |
gauge |
s (seconds) |
This metric helps you understand the load of the system to a certain extent. This metric indicates the time elapsed since the system started and the idle time of the system. |
Container metrics
Container metrics are categorized into CPU, memory, I/O, network, and other metrics.
CPU and scheduling metrics
|
Metric name |
Metric type |
Unit |
Description |
|
sysom_container_cpu_stat |
gauge |
- |
You can monitor and assess whether the ResourceQuota needs to be adjusted or other optimizations are required. This metric provides statistics on CPU resource limits in the container, including the number of periodic CPU throttling events in the cgroup, the number of times CPU throttling occurred, and the total time of CPU throttling. |
|
sysom_container_cpu_acctstat |
gauge |
% |
Shows the detailed CPU usage of the container. This metric provides the CPU usage of all tasks running in each mode in the container, including the CPU usage of the container in user mode (User), kernel mode (Kernel), and the overall CPU usage. |
|
sysom_container_cpu_cfsquota |
gauge |
- |
Shows the running time of the container limited by the CFS scheduler. This metric provides the time that the container can run in each CFS scheduling time window, including cfs_period_us and cfs_quota_us.
|
Memory metrics
|
Metric name |
Metric type |
Unit |
Description |
|
sysom_container_memory_stat |
gauge |
KiB |
Shows the usage of various types of memory in the container. This metric provides statistics on the memory usage of the container, including the usage of the container's total memory (Total), free memory (Free), available memory (Available), page cache (Cache), buffer cache (Buffers), shared memory (Shmem), reclaimable memory (SReclaimable), and unreclaimable memory (SUnreclaim). |
|
sysom_container_memory_filecache |
gauge |
KiB |
This metric helps you quickly understand the file cache usage in the container, so that you can locate memory shortage and memory latency jitter issues caused by excessive file cache usage in the container. This metric provides statistics on the usage of the file cache (Page Cache) in the container, including the top 10 files that occupy the most file cache in each container, along with their sizes and the amount of file cache they occupy. |
|
sysom_container_memory_gdrcm_latency |
gauge |
count |
Shows whether the container has latency counts and latency time caused by direct memory reclaim, where the direct memory reclaim is caused by node memory pressure. This metric provides statistics on the latency time and count caused by direct memory reclaim in the container, including the number of latencies of 1 to 5 milliseconds, 5 to 10 milliseconds, 10 to 100 milliseconds, 100 to 500 milliseconds, 500 to 1000 milliseconds, and greater than 1000 milliseconds. |
|
sysom_container_memory_cdrcm_latency |
gauge |
count |
Shows whether the container has latency counts and latency time caused by direct memory reclaim, where the direct memory reclaim is caused by pressure on the current memory control group (Memory CGroup). Note
This metric is meaningful only when the current memory control group is a non-root (Non-Root) control group, or when a memory limit is set for the current memory control group. This metric provides statistics on the latency time and count caused by direct memory reclaim in the container, including the number of direct memory reclaim latencies of 1 to 5 milliseconds, 5 to 10 milliseconds, 10 to 100 milliseconds, 100 to 500 milliseconds, 500 to 1000 milliseconds, and greater than 1000 milliseconds. |
|
sysom_container_memory_cpt_latency |
gauge |
count |
Shows whether there is latency caused by kernel memory compaction when processes in the container request memory, due to node memory pressure or excessive memory fragments. This metric provides statistics on the latency count and latency time caused by direct memory compaction, including the number of latencies of 1 to 5 milliseconds, 5 to 10 milliseconds, 10 to 100 milliseconds, 100 to 500 milliseconds, 500 to 1000 milliseconds, and greater than 1000 milliseconds. |
I/O metrics
|
Metric name |
Metric type |
Unit |
Description |
|
sysom_container_blkio_stat |
gauge |
- |
Shows the basic I/O status of the container. This metric provides the I/O statistics of the container on the corresponding disk, including the number of read/write I/O requests initiated by the container on the corresponding disk, the number of requested bytes, the number of read/write I/O requests entering the queue, the number of requested bytes, and the read/write I/O wait time. |
Network metrics
|
Metric name |
Metric type |
Unit |
Description |
|
sysom_container_network_stat |
gauge |
- |
Shows the basic network traffic of the container. This metric provides the traffic statistics of the container on the virtual NIC, including the number of packets sent/received by the container on the virtual NIC, the number of bytes, and the number of dropped packets (packet drops only include those dropped by the virtual NIC device layer, not those dropped by the network protocol stack). |