Tab | Description |
|
Overview
|
This tab provides statistics on Total Compute Nodes, Shared Storage Space, Total Jobs, and Average Load. You can also view the Cluster Status and resource usage of all compute nodes.
The following table describes the Scheduling Status of compute nodes.
-
Idle: All cores on the node are available. The node can accept new jobs and is ready to operate at its full capacity.
-
Running: Some cores on the node are in use, but others can accept new jobs.
-
Allocated: All cores on the node are in use. The node cannot accept new jobs. New jobs submitted to this node are queued until cores become available.
-
Offline: The node is not participating in computations and does not accept new jobs.
|
|
Node Monitoring
|
You can filter by cluster or queue to view the following six statistics:
-
CPU Utilization
The percentage of CPU time that programs use on compute nodes over a specific period. High utilization (over 80%–90%) generally indicates efficient resource use, but sustained high utilization can degrade cluster performance and increase response times. Low utilization (0%–30%) may suggest that resources are underutilized or that there are performance bottlenecks or job scheduling issues.
-
Memory Utilization
This metric reflects the memory utilization of compute nodes in the cluster. To ensure that jobs run smoothly, regularly monitor memory utilization to prevent job failures from out-of-memory errors. If necessary, you can resolve these issues by increasing memory capacity or optimizing memory usage strategies.
-
System Load
The workloads of compute nodes. The information helps you evaluate the job carrying capacity of the cluster. You must monitor the ratio of the load to the number of nodes. An excessively high load indicates that resources are insufficient. You can reduce system load by adding nodes (scale-out) or optimizing the workflow.
-
Disk Usage
The usage of disk storage space. The information helps you ensure disk availability and prevent excessive disk usage. For example, if the disk usage reaches 100%, write operations cannot be performed. You can effectively cope with storage pressure by clearing redundant data or expanding storage capacity. This analysis helps you get a complete picture of disk usage and determine whether space cleanup or storage expansion is required.
-
Disk Read and Write
The disk read rate indicates the amount of data read per second. The disk write rate indicates the amount of data written per second. Unit: KB/s. The metrics help you effectively evaluate the cluster performance and ensure that the cluster processes computing tasks with high efficiency.
-
Network Traffic
The amount of data transferred between compute nodes over the virtual private network (VPC). Pay special attention to periods during which the network traffic is at its highest to detect exceptions in a timely manner. For example, insufficient network bandwidth leads to lower data transmission efficiency.
|
|
Storage Monitoring
|
You can view the storage monitoring information of different file systems in the cluster, including the following six statistics:
-
Storage Space
The storage usage of the Apsara File Storage NAS (NAS) file system attached to the cluster. The NAS file system can store computing output, temporary files, and application data. To prevent exceptions such as insufficient storage space, performance degradation, and data loss, we recommend that you take measures such as monitoring and alerting, regular data cleansing, and storage expansion.
-
Files
The total number of files in the NAS file system. The metric indirectly reflects the system health status and maintenance requirements, and helps you evaluate storage management efficiency. Note that an excessive number of files increases management complexity and affects retrieval efficiency and performance. If the number of files exceeds the preset threshold, you must pay close attention to it. We recommend that you regularly organize the storage content and clear unnecessary or redundant files to improve management efficiency and system performance.
IOPS The average read and write IOPS of the file system within a specified period of time. Unit: requests/s. -
Latency
The average read and write latency per millisecond of the file system within a specific period of time. Unit: milliseconds.
-
Throughput
The average read and write throughput (bytes per second) of the file system within a specific period of time. Unit: KiB.
-
Metadata QPS
The average number of times that the file system requests metadata per second within a specific period of time. Unit: requests/s.
For more information about the storage metrics of a file system, see Performance monitoring and FAQ about the performance of NAS file systems.
|
|
Job Monitoring
|
You can filter by cluster, queue, project, or user to view the following six statistics:
-
Jobs
The number of jobs running in the cluster. It is an important metric for evaluating the cluster load. To prevent resource competition caused by an abnormally high number of jobs, make sure that the metric match with the available resources. To improve overall resource utilization efficiency, we recommend that you optimize job scheduling and eliminate unnecessary jobs.
-
Total Cores Required by Enqueued Jobs
The total number of CPU cores that are required to complete a specific task. If the total number of required CPU cores is higher than the number of available CPU cores, the waiting time of queued jobs may be long. To resolve such issues, you can increase the number of CPU cores for the cluster or adjust the priority of jobs to optimize resource allocation and improve efficiency.
-
Job Wait Duration
The average waiting time of queued jobs in the cluster. This metric reflects the situation of resource competition. A long waiting time affects the overall efficiency, especially in abnormal situations. If the waiting time increases significantly, you can run high-priority jobs first to optimize resource utilization.
-
Total Cores Used by Running Jobs
The number of cores currently being used by running jobs. This metric helps you monitor resource usage. Ensure that cores are used effectively to avoid abnormally high or low resource utilization. To resolve such issues, we recommend adjusting job configurations and allocating resources efficiently to improve overall performance.
-
Job CPU Utilization
The ratio of cores a job uses to the cores it requested. Low job CPU utilization may indicate an issue, such as an I/O bottleneck or an inefficient application, which leads to wasted resources. We recommend analyzing the job and its workload to identify opportunities for optimization.
-
Job Memory Usage
The ratio of memory a job uses to the memory it requested. Monitor job memory utilization to avoid excessive memory usage, which can lead to out-of-memory errors. To maintain system stability, you can optimize memory configurations or upgrade to a node type with more memory.
|
