ACS exposes Prometheus metrics for GPU-HPN nodes at the node level. These metrics cover CPU, memory, disk, network, and GPU subsystems. Use them to build Grafana dashboards and PromQL alerts for your GPU workloads.
Metrics
Metrics are grouped by subsystem. The Type column indicates whether a metric is a counter or a gauge: use irate() or rate() with counters; read gauges directly.
CPU
| Metric | Type | Description | Labels | Example |
|---|---|---|---|---|
node_cpu_seconds_total |
counter | Total CPU time consumed on the node, in seconds. Counter values reset to zero when the ACS monitoring component is upgraded or when a fault migration occurs during the GPU-HPN node lifecycle. Use irate() to compute per-second rates. For threshold-based alerts on this metric, add label filters to suppress false alarms caused by resets. |
NodeName: node name (spec.nodeName)<br>instance: node name (spec.nodeName)<br>mode: time-slice type — idle, iowait, irq, nice, softirq, steal, system, or user |
node_cpu_seconds_total{NodeName="cn-wulanchabu-c.cr-xxx",instance="cn-wulanchabu-c.cr-xxx",mode="user"} 135268.20999999988 |
Memory
| Metric | Type | Description | Labels | Example |
|---|---|---|---|---|
node_memory_MemAvailable_bytes |
gauge | Available memory on the node, in bytes. | NodeName: node name (spec.nodeName)<br>instance: node name (spec.nodeName) |
node_memory_MemAvailable_bytes{NodeName="cn-wulanchabu-c.cr-xxx",instance="cn-wulanchabu-c.cr-xxx"} 1.070595100672e+12 |
node_memory_MemFree_bytes |
gauge | Free memory on the node, in bytes. | NodeName: node name (spec.nodeName)<br>instance: node name (spec.nodeName) |
node_memory_MemFree_bytes{NodeName="cn-wulanchabu-c.cr-xxx",instance="cn-wulanchabu-c.cr-xxx"} 1.069967446016e+12 |
node_memory_MemTotal_bytes |
gauge | Total memory on the node, in bytes. | NodeName: node name (spec.nodeName)<br>instance: node name (spec.nodeName) |
node_memory_MemTotal_bytes{NodeName="cn-wulanchabu-c.cr-xxx",instance="cn-wulanchabu-c.cr-xxx"} 1.9327352832e+12 |
Disk
| Metric | Type | Description | Labels | Example |
|---|---|---|---|---|
node_disk_read_bytes_total |
counter | Total bytes read from the node's disks. | NodeName: node name (spec.nodeName)<br>instance: node name (spec.nodeName) |
node_disk_read_bytes_total{NodeName="cn-wulanchabu-c.cr-xxx",instance="cn-wulanchabu-c.cr-xxx"} 1.36580096e+08 |
node_disk_reads_completed_total |
counter | Total completed disk read operations on the node. | NodeName: node name (spec.nodeName)<br>instance: node name (spec.nodeName) |
node_disk_reads_completed_total{NodeName="cn-wulanchabu-c.cr-xxx",instance="cn-wulanchabu-c.cr-xxx"} 2530 |
node_disk_writes_completed_total |
counter | Total completed disk write operations on the node. | NodeName: node name (spec.nodeName)<br>instance: node name (spec.nodeName) |
node_disk_writes_completed_total{NodeName="cn-wulanchabu-c.cr-xxx",instance="cn-wulanchabu-c.cr-xxx"} 85965 |
node_disk_written_bytes_total |
counter | Total bytes written to the node's disks. | NodeName: node name (spec.nodeName)<br>instance: node name (spec.nodeName) |
node_disk_written_bytes_total{NodeName="cn-wulanchabu-c.cr-xxx",instance="cn-wulanchabu-c.cr-xxx"} 7.331622912e+09 |
Network
| Metric | Type | Description | Labels | Example |
|---|---|---|---|---|
node_network_receive_bytes_total |
counter | Total bytes received by the node. | NodeName: node name (spec.nodeName)<br>instance: node name (spec.nodeName) |
node_network_receive_bytes_total{NodeName="cn-wulanchabu-c.cr-xxx",instance="cn-wulanchabu-c.cr-xxx"} 4.5447566e+07 |
node_network_transmit_bytes_total |
counter | Total bytes transmitted by the node. | NodeName: node name (spec.nodeName)<br>instance: node name (spec.nodeName) |
node_network_transmit_bytes_total{NodeName="cn-wulanchabu-c.cr-xxx",instance="cn-wulanchabu-c.cr-xxx"} 8.6421368e+07 |
GPU
| Metric | Type | Description | Labels | Example |
|---|---|---|---|---|
DCGM_FI_DEV_COUNT |
gauge | Number of GPU devices on the node. | NodeName: node name (spec.nodeName)<br>instance: node name (spec.nodeName) |
DCGM_FI_DEV_COUNT{NodeName="cn-wulanchabu-c.cr-xxx",instance="cn-wulanchabu-c.cr-xxx"} 8 |
DCGM_FI_DEV_FB_TOTAL |
gauge | Total frame buffer capacity on the node, in MB. | NodeName: node name (spec.nodeName)<br>instance: node name (spec.nodeName) |
DCGM_FI_DEV_FB_TOTAL{NodeName="cn-wulanchabu-c.cr-xxx",instance="cn-wulanchabu-c.cr-xxx"} 1.56672e+06 |
DCGM_FI_DEV_FB_USED |
gauge | Frame buffer used per GPU device, in MB. | NodeName: node name (spec.nodeName)<br>instance: node name (spec.nodeName)<br>UUID: unique device identifier<br>modelName: device model name<br>device: device name<br>gpu: device number |
DCGM_FI_DEV_FB_USED{NodeName="cn-wulanchabu-c.cr-xxx",UUID="GPU-hashID",instance="cn-wulanchabu-c.cr-xx",modelName="mode-name-demo"} 9672 |
DCGM_FI_DEV_GPU_UTIL |
gauge | GPU utilization per device, in percent (0–100). | NodeName: node name (spec.nodeName)<br>instance: node name (spec.nodeName)<br>UUID: unique device identifier<br>modelName: device model name<br>device: device name<br>gpu: device number |
DCGM_FI_DEV_GPU_UTIL{NodeName="cn-wulanchabu-c.cr-xxx",UUID="GPU-hashID",instance="cn-wulanchabu-c.cr-xx",modelName="mode-name-demo"} 56 |
sysom_imc_node_event |
gauge | Node-level memory bandwidth performance, aggregated across multiple NUMA sockets. Collected every 30 seconds. | instance: node name (spec.nodeName)<br>value: metric subtype — bw_rd (read bandwidth, in MB/s), bw_wr (write bandwidth, in MB/s), rlat (average read latency, in ns). Other subtypes are not currently supported. |
sysom_imc_node_event{instance="cn-wulanchabu-c.cr-akrjaz1r0csm2qdrk227",value="bw_rd"} 780 |
Node lifecycle
| Metric | Type | Description | Labels | Example |
|---|---|---|---|---|
node_boot_time_seconds |
gauge | Timestamp recorded when the GPU-HPN node is provisioned. Updated to the completion time of the most recent auto repair event when the node triggers auto repair due to a failure. | None | node_boot_time_seconds 1.735635132e+09 |
Usage notes
Timestamps
Each metric carries a timestamp attribute in standard Prometheus text exposition format, indicating when the resource metric was collected:
node_cpu_seconds_total{NodeName="cn-wulanchabu-c.cr-xxx",instance="cn-wulanchabu-c.cr-xxx",mode="idle"} 17.509999999999998 1735112457237Use this with the honor_timestamps configuration in Prometheus. The built-in Prometheus dashboard in ACS has honor_timestamps enabled by default.
Counter resets
GPU-HPN nodes in ACS are not physical machines. Unlike ECS nodes where counter values are tracked by the operating system and reset on OS restart, ACS counter metrics are collected by the ACS monitoring component. Counter values reset to zero when:
-
The ACS monitoring component is changed or upgraded
-
A fault migration occurs during the GPU-HPN node lifecycle
Use irate() to compute rates from counter metrics, which is more robust to resets than rate(). For threshold-based alerts on counter metrics, add label filters to avoid false alarms.
FAQ
How do I distinguish ACS Pod metrics with the same name, such as DCGM_FI_DEV_FB_USED, when configuring a Grafana dashboard?
Pod metrics carry Namespace and Pod labels that node-level metrics do not. Filter or group by these labels in your PromQL queries to distinguish Pod-scoped metrics from node-scoped metrics with the same name.