All Products
Search

This Product

    Container Compute Service:ACS GPU pod monitoring metrics

    Document Center

    Container Compute Service:ACS GPU pod monitoring metrics

    Last Updated:Mar 26, 2026

    GPU monitoring uses an Exporter + Prometheus + Grafana architecture to provide comprehensive GPU observability. Use the metrics from the GPU Exporter for Container Service to build Grafana dashboards and set up alerts. This topic describes the GPU monitoring metrics.

    Billing

    The GPU Exporter is compatible with the open-source DCGM Exporter. The following GPU monitoring metrics are basic metrics — no additional fees are charged when using them in Prometheus. Custom metrics outside this list incur additional charges. For details, see Billing overview.

    Metrics

    DCGM metrics

    Filter DCGM metrics using the following dimensions:

    • namespace="{{pod_namespace}}"

    • pod="{{pod_name}}"

    • Hostname="{{pod_name}}"

    • NodeName="cn-wulanchabu-c.cr-xxx" (GPU-HPN pods only)

    • UUID="GPU-example-uuid-abcd"

    • device="nvidia0"

    • gpu="0"

    • modelName="example-model"

    GPU resource metrics

    Metric DCGM field name Type Unit Description
    GPU utilization DCGM_FI_DEV_GPU_UTIL Gauge % The percentage of time that one or more kernel functions are active during a sample period. The sample period is 1 second or 1/6 of a second, depending on the GPU product. This metric shows that a kernel is running on the GPU, but does not break down what the kernel is doing — use profiling metrics for deeper analysis.
    Frame buffer used DCGM_FI_DEV_FB_USED Gauge MiB The amount of used frame buffer (video memory).
    Frame buffer total DCGM_FI_DEV_FB_TOTAL Gauge MiB The total frame buffer (video memory) capacity.
    Encoder utilization DCGM_FI_DEV_ENC_UTIL Gauge % The percentage of time the hardware encoder engine is active.
    Decoder utilization DCGM_FI_DEV_DEC_UTIL Gauge % The percentage of time the hardware decoder engine is active.
    Memory bandwidth utilization DCGM_FI_DEV_MEM_COPY_UTIL Gauge % The percentage of peak memory bandwidth in use. For example, an NVIDIA V100 has a peak memory bandwidth of 900 GB/s. If the current bandwidth is 450 GB/s, this metric reports 50%.

    Profiling metrics

    Metric DCGM field name Type Unit Description
    SM activity DCGM_FI_PROF_SM_ACTIVE Gauge % The fraction of time at least one warp is active on a Streaming Multiprocessor (SM), averaged across all SMs. A warp is considered active once it is scheduled and its resources are allocated — it can be executing instructions or waiting on memory. Values below 0.5 indicate inefficient GPU use; values above 0.8 are required for high efficiency. For example, with N SMs: running N thread blocks for the full interval gives 1.0; running N/5 thread blocks, or N blocks for 1/5 of the interval, gives 0.2.
    SM occupancy DCGM_FI_PROF_SM_OCCUPANCY Gauge % The ratio of resident warps on an SM to the maximum number of warps that SM supports, averaged across all SMs. Higher occupancy does not always mean higher GPU utilization. For workloads limited by GPU memory bandwidth (DCGM_FI_PROF_DRAM_ACTIVE), higher occupancy indicates more effective GPU use.
    DRAM active DCGM_FI_PROF_DRAM_ACTIVE Gauge % The fraction of cycles that device memory (DRAM) is sending or receiving data, averaged over the interval. A value of 1.0 (100%) means a DRAM instruction executes every cycle; in practice, the achievable maximum is about 0.8 (80%). A value of 0.2 means 20% of cycles were spent on device memory reads or writes.
    NVLink receive throughput DCGM_FI_PROF_NVLINK_RX_BYTES Counter B/s The average rate of data received over NVLink, excluding protocol headers. Averaged over the interval — for example, 1 GB transferred in 1 second reports 1 GB/s regardless of burst patterns. The theoretical maximum for NVLink Gen2 is 25 GB/s per link per direction.
    NVLink transmit throughput DCGM_FI_PROF_NVLINK_TX_BYTES Counter B/s The average rate of data transmitted over NVLink, excluding protocol headers. Averaged over the interval. The theoretical maximum for NVLink Gen2 is 25 GB/s per link per direction.
    PCIe receive throughput DCGM_FI_PROF_PCIE_RX_BYTES Counter B/s The average rate of data received over the PCIe bus, including protocol headers and data payload. Averaged over the interval. The theoretical maximum for PCIe Gen3 is 985 MB/s per channel.
    PCIe transmit throughput DCGM_FI_PROF_PCIE_TX_BYTES Counter B/s The average rate of data transmitted over the PCIe bus, including protocol headers and data payload. Averaged over the interval. The theoretical maximum for PCIe Gen3 is 985 MB/s per channel.
    Tensor Core activity DCGM_FI_PROF_PIPE_TENSOR_ACTIVE Gauge % The fraction of cycles the Tensor (HMMA/IMMA) Pipe is active, averaged over the interval. A value of 1.0 means a Tensor instruction is issued every other instruction cycle (each instruction takes two cycles). A value of 0.2 can reflect different distributions: 20% of SMs running Tensor Cores at 100% for the full interval, 100% of SMs running at 20% utilization, 100% of SMs running at 100% for 1/5 of the interval, or other combinations.

    Frequency

    Metric DCGM field name Type Unit Description
    SM clock frequency DCGM_FI_DEV_SM_CLOCK Gauge MHz The current SM clock frequency.

    GPU exceptions and XID errors

    Metric DCGM field name Type Unit Description
    NVSwitch fatal error DCGM_FI_DEV_NVSWITCH_FATAL_ERRORS Gauge Error code NVSwitch fatal error information. The value is the SXid error code.
    Row remap failure DCGM_FI_DEV_ROW_REMAP_FAILURE Gauge — Indicates that a row remap error has occurred.
    Row remap pending DCGM_FI_DEV_ROW_REMAP_PENDING Gauge — Indicates that a row remap is pending.

    Temperature and power

    Metric DCGM field name Type Unit Description
    GPU temperature DCGM_FI_DEV_GPU_TEMP Gauge ℃ The GPU temperature.
    Memory temperature DCGM_FI_DEV_MEMORY_TEMP Gauge ℃ The memory temperature.
    Power consumption DCGM_FI_DEV_POWER_USAGE Gauge W The current power consumption.

    Retired pages

    Metric DCGM field name Type Unit Description
    Retired pages (SBE) DCGM_FI_DEV_RETIRED_SBE Gauge — The number of pages retired due to single-bit errors (SBE).
    Retired pages (DBE) DCGM_FI_DEV_RETIRED_DBE Gauge — The number of pages retired due to double-bit errors (DBE).

    RDMA metrics

    Filter RDMA metrics using the following dimensions:

    • app="nusa-exporter"

    • hostname="{{pod_name}}"

    • ip="172.16.17.114"

    • namespace="{{pod_namespace}}"

    • node="{{virtual-kubelet-nodename}}"

    • pod="{{pod_name}}"

    Metric Type Unit Description
    rdma_service_monitor_tx_bytes_rate Gauge bytes Instantaneous outbound traffic on the pod RDMA network.
    rdma_service_monitor_rx_bytes_rate Gauge bytes Instantaneous inbound traffic on the pod RDMA network.
    rdma_service_monitor_tx_bytes Counter bytes Cumulative outbound traffic on the pod RDMA network.
    rdma_service_monitor_rx_bytes Counter bytes Cumulative inbound traffic on the pod RDMA network.
    rdma_service_monitor_tx_packets_rate Gauge packets Instantaneous outbound packet rate on the pod RDMA network.
    rdma_service_monitor_rx_packets_rate Gauge packets Instantaneous inbound packet rate on the pod RDMA network.
    rdma_service_monitor_tx_packets Counter packets Cumulative number of outbound packets on the pod RDMA network.
    rdma_service_monitor_rx_packets Counter packets Cumulative number of inbound packets on the pod RDMA network.