Customize EAS observability dashboards and alerts using ARMS - Platform For AI

Application Real-Time Monitoring Service (ARMS) is a cloud-native observability platform provided by Alibaba Cloud. You can use ARMS to create custom observability dashboards for PAI-EAS services and configure flexible alert rules for comprehensive EAS metric monitoring.

Billing information

Using ARMS incurs charges. For detailed billing information, see ARMS billing.

Ingest EAS monitoring metrics

Log on to the or the ARMS console. In the navigation pane on the left, click Integration Center.
On the Integration Center page, click the AI tab, then click Alibaba Cloud PAI EAS Model Online Service.
On the Start Integration tab of the panel that appears, select a region for data storage, enter an integration name, then click OK.

After you click OK, the panel closes. The integration runs in the background and takes about 1–2 minutes to complete.

Important
Advanced monitoring metrics include large model inference framework metrics (vLLM or SGLang), GPU computing power metrics, PAI EAS access gateway metrics, service and tenant dimension statistics, and custom inference application metrics. Supported regions include Beijing, Shanghai, Hangzhou, Ulanqab, Singapore, and Heyuan. To enable other regions, submit a ticket. Turn on the Advanced monitoring metrics switch.

When you turn on the Advanced monitoring metrics switch, all EAS service and resource group metrics are included. If you use custom monitoring metrics, they are also included in advanced monitoring metrics, with the prefix custom_ added to their names.
To verify that the integration is successful, click Integration Management and check that your environment appears in the integrated environment list.

View ingested EAS metric data

In the navigation pane on the left, click Integration Management. On the Integrated Addons tab, click the Alibaba Cloud PAI EAS Model Online Service card.

The panel that appears shows a list of integrated environments. After clicking the card, a details panel opens on the right. On the Environment List tab, you can view integrated environments (such as cn-hangzhou, cn-beijing, cn-shanghai) and component counts. Click view details to go to the metrics page for the selected environment.

In the Actions column of the target environment, click View Details under the Metrics Explorer tab. On this page, you can view all metric details for the EAS service.

Viewing method	Description
View metric details by filtering	Cloud Monitor metric names include the prefix AliyunLearn_eas, matching the EAS metric definitions shown in Cloud Monitor, with richer label information. For advanced monitoring metrics, see EAS advanced monitoring metrics. On the Metric Explorer tab, in the query builder, enter a metric keyword (such as `AliyunLearn_eas_cpu_core_usage`) in the Metric search box. Select your target metric from the dropdown list (including CPU core usage, CPU utilization, GPU memory, and utilization). Set filter conditions using the Labels dropdown, then click Run query to view the corresponding line chart.
Use PromQL expressions to query richer metrics.	For example, to query the sum of QPS across all services, switch to Code, enter `sum(AliyunLearn_eas_qps_total)`, then click Run query. The chart displays the change trends of total QPS for all EAS services deployed in your current region over recent time. For more information about Prometheus Query Language (PromQL) syntax, see Query and analyze time series data. You can also click the AI assistant button next to the input box to learn PromQL syntax.

Customize observability dashboards

View Grafana dashboard details.

ARMS observability dashboards use Grafana and include a default Grafana dashboard. Follow these steps to view dashboard details.
1. Go to the cloud service environment details page. For specific steps, see Step 2: View monitoring dashboards.
2. On the Component Management tab, under Addon Type, select Alibaba Cloud PAI EAS Online Prediction Service. Then click Dashboards and the dashboard name to view the built-in Grafana dashboard.
Add a global QPS panel to the default Grafana dashboard.
1. On the dashboard details page, click the Add panel button in the upper-right corner. In the new Add panel panel, click Add a new panel.
2. On the Edit Panel page, on the right side, change the chart type to Stat.
3. In the lower-left corner of the page, change Data source to ${datasource}. In the Metrics browser text box, enter the PromQL query statement sum(AliyunLearn_eas_eas_qps_total), then click Run queries.
4. Adjust thresholds to assign different display colors for different values. After configuration, the page previews the chart. Click Apply to save your settings.

For more information about Grafana, see Managed Service for Grafana.

Customize monitoring alerts

ARMS provides alerting capabilities. Follow Step 2: View monitoring dashboards to go to the cloud service environment details page. On the Alert rules tab, you can view default alert rule templates. In the ARMS console, on the Provisioning page, select the corresponding cloud service region environment. In the Component Type list on the left, select Alibaba Cloud PAI EAS Online Prediction Service, then click the Alert Rules tab. The following default P2-level alert rules are preconfigured, all initially Stopped, with alert group cloud-learn_eas-Cloud-Default:

Alert rule (ARMS console display name)	PromQL metric name
Alibaba Cloud PAI EAS GPU Memory Utilization Alert	`AliyunLearn_eas_eas_gpu_memory_util`
Alibaba Cloud PAI EAS 5XX Response Ratio Alert	`AliyunLearn_eas_eas_rps_status_5xx_ratio`
Alibaba Cloud PAI EAS 4XX Response Ratio Alert	`AliyunLearn_eas_eas_rps_status_4xx_ratio`
Alibaba Cloud PAI EAS 2XX Response Ratio Alert	`AliyunLearn_eas_eas_rps_status_2xx_ratio`
Alibaba Cloud PAI EAS Memory Utilization Percentage Alert	`AliyunLearn_eas_eas_memory_util_per`
Alibaba Cloud PAI EAS Memory Utilization Alert	`AliyunLearn_eas_eas_memory_util`

When querying these metrics in Metrics Explorer, use the full PromQL metric name with the AliyunLearn_eas_eas_ prefix. For example, to query GPU memory utilization, enter AliyunLearn_eas_eas_gpu_memory_util{} > 80.

If these default templates do not meet your needs, follow these steps to configure custom alert rules.

Log on to the or the ARMS console. In the navigation pane on the left, choose Managed Service for Prometheus > Prometheus Alert Rules, then click Create Prometheus Alert Rule.

On the Create Prometheus Alert Rule page, configure the following key parameters. For more parameter details, see Create Prometheus alert rules.

Parameter	Description
Check Type	Select Custom PromQL.
Custom PromQL statement	Enter `sum(AliyunLearn_eas_eas_qps_total) > 20`.
Alert Message	The alert information recipients receive.
Alert Notification	Set notification recipients.

Click Completed.

You can view your created alert rule on the Prometheus Alert Rules page. When the total global QPS across all services exceeds 20, your configured notification recipients receive an alert.

Appendix: EAS advanced monitoring metrics

Note

The following metrics appear only when ARMS advanced monitoring metrics are enabled.

Metric	Description	Labels (Dimensions)	Category	Type	Unit	Period (seconds)
instance_cpu_count	CPU count per service instance	instance,resource_type	CPU	Gauge	count	60
instance_gpu_count	GPU count per service instance	instance,resource_type	GPU	Gauge	count	60
instance_cpu_usage	CPU usage per service instance	instance	CPU	Gauge	core	60
instance_user_cpu_usage	User process CPU usage per service instance	instance	CPU	Gauge	core	60
instance_system_cpu_usage	System process CPU usage per service instance	instance	CPU	Gauge	core	60
instance_cpu_util	CPU utilization per service instance	instance	CPU	Gauge	%	60
instance_memory_rss_usage	Memory usage per service instance	instance	Memory	Gauge	byte	60
instance_memory_cache_usage	Memory cache usage per service instance	instance	Memory	Gauge	byte	60
instance_memory_total	Total memory per service instance	instance	Memory	Gauge	byte	60
instance_memory_util	Memory utilization per service instance	instance	Memory	Gauge	%	60
instance_response	Request count per service instance	instance	Request	Counter	count	60
instance_gpu_util	GPU utilization per service instance	instance	GPU	Gauge	%	60
instance_gpu_memory_usage	GPU memory usage per service instance	instance	GPU	Gauge	MiB	60
instance_gpu_memory_total	Total GPU memory per service instance	instance	GPU	Gauge	MiB	60
instance_gpu_memory_util	GPU memory utilization per service instance	instance	GPU	Gauge	MiB	60
instance_gpu_memory_bandwidth_limit	GPU memory bandwidth limit per service instance	instance	GPU	Gauge	bytes/second	60
instance_gpu_temperature	GPU temperature per service instance	instance	GPU	Gauge	°C	60
instance_gpu_slow_temperature	GPU slowdown temperature per service instance	instance	GPU	Gauge	°C	60
instance_gpu_shut_temperature	GPU shutdown temperature per service instance	instance	GPU	Gauge	°C	60
instance_gpu_nvswitch_error	Fatal NVSwitch error information per service instance	instance	GPU	Gauge	count	60 s
instance_gpu_nvswitch_non_fatal_error	Non-fatal NVSwitch error information per service instance	instance	GPU	Gauge	count	60
instance_gpu_ecc_total_vol_sbe	Total single-bit volatile ECC errors per service instance	instance	GPU	Counter	count	60
instance_gpu_ecc_total_vol_dbe	Total double-bit volatile ECC errors per service instance	instance	GPU	Counter	count	60
instance_gpu_ecc_total_agg_sbe	Total single-bit aggregated (persistent) ECC errors per service instance	instance	GPU	Counter	count	60
instance_gpu_ecc_total_agg_dbe	Total double-bit aggregated (persistent) ECC errors per service instance	instance	GPU	Counter	count	60
instance_gpu_remap_fail	Row remap failure count per service instance	instance	GPU	Gauge	count	60
instance_gpu_remap_pending	Pending row remap count per service instance	instance	GPU	Gauge	count	60
instance_gpu_pcie_replay_counter	PCIe replay counter per service instance	instance	GPU	Gauge	count	60
instance_gpu_pcie_transmit_measure_by_dcgm	PCIe transmit rate measured by DCGM per service instance	instance	GPU	Gauge	bytes/second	60
instance_gpu_pcie_receive_measure_by_dcgm	PCIe receive rate measured by DCGM per service instance	instance	GPU	Gauge	bytes/second	60
instance_gpu_graphics_engine_util	Graphics engine utilization per service instance	instance	GPU	Gauge	ratio (0~1)	60
instance_gpu_sm_util	SM (streaming multiprocessor) utilization per service instance	instance	GPU	Gauge	ratio (0~1)	60
instance_gpu_dram_active	Ratio of active data transmission or reception on device memory interface per service instance	instance	GPU	Gauge	ratio (0~1)	60
instance_gpu_tensortflops_used	Tflops used by Tensor pipeline per service instance	instance	GPU	Gauge	count	60
instance_gpu_memory_bandwidth_used	Memory bandwidth usage per service instance	instance	GPU	Gauge	bytes/second	60
instance_gpu_sm_clock	SM clock frequency per service instance	instance	GPU	Gauge	MHz	60
instance_gpu_sm_occupancy	Ratio of resident Warp threads on SM per service instance	instance	GPU	Gauge	ratio (0~1)	60
instance_gpu_fp32tflops_used	Tflops used by FP32 pipeline per service instance	instance	GPU	Gauge	count	60
instance_gpu_fp16tflops_used	Tflops used by FP16 pipeline per service instance	instance	GPU	Gauge	count	60
instance_gpu_pipe_fp32_active	Active cycle ratio of FP32 pipeline per service instance	instance	GPU	Gauge	ratio (0~1)	60
instance_gpu_pipe_fp16_active	Active cycle ratio of FP16 pipeline per service instance	instance	GPU	Gauge	ratio (0~1)	60 s
instance_gpu_pipe_tensor_active	Active cycle ratio of Tensor pipeline per service instance	instance	GPU	Gauge	ratio (0~1)	60
instance_gpu_power_usage	GPU power consumption per service instance	instance	GPU	Gauge	watts	60
instance_accelerator_power_usage	Accelerator power consumption per service instance	instance	GPU	Gauge	milliwatts	60
instance_gpu_mem_copy_util	Memory copy utilization per service instance	instance	GPU	Gauge	%	60
instance_gpu_health_count	Total GPU health status count per service instance	instance	GPU	Gauge	count	60
instance_gpu_lost_card_num	Number of lost GPUs in VM per service instance	instance	GPU	Gauge	count	60
instance_gpu_driver_hang	Driver hang count per service instance	instance	GPU	Gauge	count	60
instance_gpu_profile_status	Amperf performance analysis status per service instance	instance	GPU	Gauge	count	60
instance_gpu_uncorrectable_ecc	Uncorrectable ECC error count per service instance	instance	GPU	Gauge	count	60
instance_gpu_xid_cnt	Xid error count per service instance	instance	GPU	Gauge	count	60
instance_gpu_fatal_xid_error	Fatal Xid error count per service instance	instance	GPU	Gauge	count	60
instance_gpu_kernel_err_cnt	Non-Xid error count from kernel logs per service instance	instance	GPU	Gauge	count	60
instance_qps	Requests per second per service instance	instance	Request	Gauge	count	60
instance_traffic	Traffic per service instance	instance	Request	Gauge	bps	60
instance_avg_latency	Average request response time per service instance	instance	Request	Gauge	ms	60
instance_tpxx_latency	TOPXX request response time per service instance	instance	Request	Gauge	ms	60
instance_traffic_in	Inbound traffic per service instance	instance	Request	Gauge	bps	60
instance_traffic_out	Outbound traffic per service instance	instance	Request	Gauge	bps	60
instance_tcp_connections	TCP connection count per service instance	instance	Request	Gauge	count	60
service_replicas	Service instance count	service	Meta	Gauge	count	60
service_pending_replicas	Pending service instance count	service	Meta	Gauge	count	60
service_available_replicas	Running service instance count	service	Meta	Gauge	count	60
service_replicas_with_resource_type	Service instance count (with resource type label)	service	Meta	Gauge	count	60
service_cpu_count	Total CPU cores used by service	service	CPU	Gauge	core	60
service_cpu_count_with_resource_type	Total CPU cores used by service (with resource type label)	service	CPU	Gauge	core	60
service_gpu_count_with_resource_type	Total GPUs used by service (with resource type label)	service	GPU	Gauge	count	60
service_rps_status_2xx	2XX response request count for service	service	Request	Gauge	count	60
service_rps_status_4xx	4XX response request count for service	service	Request	Gauge	count	60
service_rps_status_5xx	5XX response request count for service	service	Request	Gauge	count	60
service_rps_status_2xx_ratio	2XX response request ratio for service	service	Request	Gauge	%	60
service_rps_status_4xx_ratio	4XX response request ratio for service	service	Request	Gauge	%	60
service_rps_status_5xx_ratio	5XX response request ratio for service	service	Request	Gauge	%	60
service_qps	Requests per second for service	service	Request	Gauge	count	60
service_avg_latency	Average request response time for service	service	Request	Gauge	ms	60
service_tpxx_latency	TOPXX request response time for service	service	Request	Gauge	ms	60
service_tp100_latency	TOP100 request response time for service	service	Request	Gauge	ms	60
service_traffic_in	Inbound traffic for service	service	Network	Gauge	bps	60
service_traffic_out	Outbound traffic for service	service	Network	Gauge		60
service_cpu_usage	CPU usage for service	service	CPU	Gauge	core	60
service_user_cpu_usage	User process CPU usage for service	service	CPU	Gauge	core	60
service_system_cpu_usage	System process CPU usage for service	service	CPU	Gauge	core	60
service_cpu_util	CPU utilization for service	service	CPU	Gauge	%	60
service_memory_rss_usage	Memory usage for service	service	Memory	Gauge	byte	60
service_memory_cache_usage	Memory cache usage for service	service	Memory	Gauge	byte	60
service_memory_total	Total memory for service	service	Memory	Gauge	byte	60
service_memory_util	Memory utilization for service	service	Memory	Gauge	%	60
service_gpu_util	GPU utilization for service	service	GPU	Gauge	%	60
service_gpu_memory_usage	GPU memory usage for service	service	GPU	Gauge	MiB	60
service_gpu_memory_total	Total GPU memory for service	service	GPU	Gauge	MiB	60
service_gpu_memory_util	GPU memory utilization for service	service	GPU	Gauge	MiB	60
service_gpu_memory_bandwidth_limit	GPU memory bandwidth limit for service	service	GPU	Gauge	bytes/second	60
service_gpu_temperature	GPU temperature for service	service	GPU	Gauge	°C	60
service_gpu_slow_temperature	GPU slowdown temperature for service	service	GPU	Gauge	°C	60
service_gpu_shut_temperature	GPU shutdown temperature for service	service	GPU	Gauge	°C	60
service_gpu_nvswitch_error	Fatal NVSwitch error information for service	service	GPU	Gauge	count	60
service_gpu_nvswitch_non_fatal_error	Non-fatal NVSwitch error information for service	service	GPU	Gauge	count	60
service_gpu_ecc_total_vol_sbe	Total single-bit volatile ECC errors for service	service	GPU	Counter	count	60
service_gpu_ecc_total_vol_dbe	Total double-bit volatile ECC errors for service	service	GPU	Counter	count	60
service_gpu_ecc_total_agg_sbe	Total single-bit aggregated (persistent) ECC errors for service	service	GPU	Counter	count	60
service_gpu_ecc_total_agg_dbe	Total double-bit aggregated (persistent) ECC errors for service	service	GPU	Counter	count	60
service_gpu_remap_fail	Row remap failure count for service	service	GPU	Gauge	count	60
service_gpu_remap_pending	Pending row remap count for service	service	GPU	Gauge	count	60
service_gpu_pcie_replay_counter	PCIe replay counter for service	service	GPU	Gauge	count	60
service_gpu_pcie_transmit_measure_by_dcgm	PCIe transmit rate measured by DCGM for service	service	GPU	Gauge	bytes/second	60
service_gpu_pcie_receive_measure_by_dcgm	PCIe receive rate measured by DCGM for service	service	GPU	Gauge	bytes/second	60
service_gpu_graphics_engine_util	Graphics engine utilization for service	service	GPU	Gauge	ratio (0~1)	60
service_gpu_sm_util	SM (streaming multiprocessor) utilization for service	service	GPU	Gauge	ratio (0~1)	60
service_gpu_dram_active	Ratio of active data transmission or reception on device memory interface for service	service	GPU	Gauge	ratio (0~1)	60
service_gpu_tensortflops_used	Tflops used by Tensor pipeline for service	service	GPU	Gauge	count	60
service_gpu_memory_bandwidth_used	Memory bandwidth usage for service	service	GPU	Gauge	bytes/second	60
service_gpu_sm_clock	SM clock frequency for service	service	GPU	Gauge	MHz	60
service_gpu_sm_occupancy	Ratio of resident Warp threads on SM for service	service	GPU	Gauge	ratio (0~1)	60
service_gpu_fp32tflops_used	Tflops used by FP32 pipeline for service	service	GPU	Gauge	count	60
service_gpu_fp16tflops_used	Tflops used by FP16 pipeline for service	service	GPU	Gauge	count	60
service_gpu_pipe_fp32_active	Active cycle ratio of FP32 pipeline for service	service	GPU	Gauge	ratio (0~1)	60
service_gpu_pipe_fp16_active	Active cycle ratio of FP16 pipeline for service	service	GPU	Gauge	ratio (0~1)	60
service_gpu_pipe_tensor_active	Active cycle ratio of Tensor pipeline for service	service	GPU	Gauge	ratio (0~1)	60
service_gpu_power_usage	GPU power consumption for service	service	GPU	Gauge	watts	60
service_accelerator_power_usage	Accelerator power consumption for service	service	GPU	Gauge	milliwatts	60
service_gpu_mem_copy_util	Memory copy utilization for service	service	GPU	Gauge	%	60
service_gpu_health_count	Total GPU health status count for service	service	GPU	Gauge	count	60
service_gpu_lost_card_num	Number of lost GPUs in VM for service	service	GPU	Gauge	count	60
service_gpu_driver_hang	Service Driver Suspension Count	service	GPU	Gauge	count	60
service_gpu_profile_status	Amperf performance analysis status for service	service	GPU	Gauge	count	60
service_gpu_uncorrectable_ecc	Uncorrectable ECC error count for service	service	GPU	Gauge	count	60
service_gpu_xid_cnt	Xid error count for service	service	GPU	Gauge	count	60
service_gpu_fatal_xid_error	Fatal Xid error count for service	service	GPU	Gauge	count	60
service_gpu_kernel_err_cnt	Non-Xid error count from kernel logs for service	service	GPU	Gauge	count	60
service_tcp_connections	TCP connection count for service	service	Network	Gauge	count	60
service_gateway_requests	llm-gateway: current requests received by gateway	service	Request	Gauge	count	60
service_gateway_pending_requests	llm-gateway: current requests cached in gateway	service	Request	Gauge	count	60
service_llm_ttft_max	llm-gateway: maximum time to first token for LLM streaming requests	service	Request	Gauge	time	60
service_llm_ttft_min	llm-gateway: minimum time to first token for LLM streaming requests	service	Request	Gauge	time	60
service_llm_ttft_mean	llm-gateway: average time to first token for LLM streaming requests	service	Request	Gauge	time	60
service_llm_ttft_percent	llm-gateway: percentile time to first token for LLM streaming requests	service	Request	Gauge	time	60
service_llm_tpot_max	llm-gateway: maximum time per output token for LLM streaming requests	service	Request	Gauge	time	60
service_llm_tpot_min	llm-gateway: minimum time per output token for LLM streaming requests	service	Request	Gauge	time	60
service_llm_tpot_mean	llm-gateway: average time per output token for LLM streaming requests	service	Request	Gauge	time	60
service_llm_tpot_percent	llm-gateway: percentile time per output token for LLM streaming requests	service	Request	Gauge	time	60
service_endpoint_llm_waiting_requests	llm-gateway: number of requests queued in LLM inference engine	service	Request	Gauge	count	60
service_endpoint_llm_running_requests	llm-gateway: number of requests being processed in LLM inference engine	service	Request	Gauge	count	60
service_endpoint_llm_gpu_cache_usage	llm-gateway: GPU KV-cache utilization in LLM inference engine	service	Request	Gauge	count	60
service_endpoint_llm_tps_in	llm-gateway: input tokens per second in LLM engine	service	Request	Gauge	count	60
service_endpoint_llm_tps_out	llm-gateway: output tokens per second in LLM engine	service	Request	Gauge	count	60
resource_instance_cpu_util	CPU utilization per resource group instance	instance_id	Resource Instance	Gauge	%	60
resource_instance_memory_total	Total memory per resource group instance	instance_id	Resource Instance	Gauge	byte	60
resource_instance_memory_used	Memory usage per resource group instance	instance_id	Resource Instance	Gauge	byte	60
resource_instance_memory_util	Memory utilization per resource group instance	instance_id	Resource Instance	Gauge	%	60
resource_instance_memory_cache	Memory cache usage per resource group instance	instance_id	Resource Instance	Gauge	byte	60
resource_instance_memory_free	Free memory per resource group instance	instance_id	Resource Instance	Gauge	byte	60
resource_instance_traffic_in	Inbound traffic per resource group instance	instance_id	Resource Instance	Gauge	bytes/second	60
resource_instance_traffic_out	Outbound traffic per resource group instance	instance_id	Resource Instance	Gauge	bytes/second	60
resource_instance_disk_used	Disk usage per resource group instance	instance_id	Resource Instance	Gauge	byte	60
resource_instance_disk_total	Total disk per resource group instance	instance_id	Resource Instance	Gauge	byte	60
resource_instance_disk_util	Disk utilization per resource group instance	instance_id	Resource Instance	Gauge	byte	60
resource_instance_tcp_established	Established TCP connections per resource group instance	instance_id	Resource Instance	Gauge	count	60
resource_instance_tcp_time_wait	TIME_WAIT TCP connections per resource group instance	instance_id	Resource Instance	Gauge	count	60
resource_instance_gpu_util	GPU utilization per resource group instance	instance_id	Resource Instance	Gauge	%	60
resource_instance_gpu_memory_usage	GPU memory usage per resource group instance	instance_id	Resource Instance	Gauge	MiB	60
resource_instance_gpu_memory_total	Total GPU memory per resource group instance	instance_id	Resource Instance	Gauge	MiB	60
resource_instance_gpu_memory_util	GPU memory utilization per resource group instance	instance_id	Resource Instance	Gauge	%	60
resource_cpu_util	CPU utilization per resource group	resource	Resource	Gauge	%	60
resource_memory_total	Total memory per resource group	resource	Resource	Gauge	byte	60
resource_memory_used	Memory usage per resource group	resource	Resource	Gauge	byte	60
resource_memory_util	Memory utilization per resource group	resource	Resource	Gauge	%	60
resource_memory_cache	Memory cache usage per resource group	resource	Resource	Gauge	byte	60
resource_memory_free	Free memory per resource group	resource	Resource	Gauge	byte	60
resource_traffic_in	Inbound traffic per resource group	resource	Resource	Gauge	bytes/second	60
resource_traffic_out	Outbound traffic per resource group	resource	Resource	Gauge	bytes/second	60
resource_disk_used	Disk usage per resource group	resource	Resource	Gauge	byte	60
resource_disk_total	Total disk per resource group	resource	Resource	Gauge	byte	60
resource_disk_util	Disk utilization per resource group	resource	Resource	Gauge	byte	60
resource_tcp_established	Established TCP connections per resource group	resource	Resource	Gauge	count	60
resource_tcp_time_wait	TIME_WAIT TCP connections per resource group	resource	Resource	Gauge	count	60
resource_gpu_util	GPU utilization per resource group	resource	Resource	Gauge	%	60
resource_gpu_memory_usage	GPU memory usage per resource group	resource	Resource	Gauge	MiB	60
resource_gpu_memory_total	Total GPU memory per resource group	resource	Resource	Gauge	MiB	60
resource_gpu_memory_util	GPU memory utilization per resource group	resource	Resource	Gauge	%	60