Resource quotas allow you to monitor resource status by providing metrics for GPU, CPU, and memory. After you configure an alert rule, the system sends a notification when a resource metric is abnormal, for example, when CPU utilization exceeds a threshold. You can use CloudMonitor and ARMS to view monitoring data, configure alerts, and subscribe to metrics.
Prerequisites
Before you set up monitoring and alerting, you must create a resource quota. For more information, see Introduction to resource quotas.
Metrics
PAI-Quota provides metrics for GPU, CPU, memory, disk, and network resources. You can view these metrics by quota or by node. For a complete list and detailed descriptions of all metrics, see PAI-Quota metrics.
By quota
Metric | Description |
GPU compute utilization (by quota) | The GPU compute utilization for the quota. |
GPU memory utilization (by quota) | The GPU memory utilization for the quota. |
Scheduled GPUs (by quota) | The number of scheduled GPUs for the quota. |
Total GPUs (by quota) | The total number of GPUs for the quota. |
GPU device power (by quota) | The GPU device power consumption for the quota. |
Scheduled CPUs (by quota) | The number of scheduled CPU cores for the quota. |
Total CPUs (by quota) | The total number of CPU cores for the quota. |
CPU utilization (by quota) | The CPU utilization percentage for the quota. |
Memory utilization (by quota) | The memory utilization percentage for the quota. |
For more metrics, see PAI-Quota metrics. | |
By node
Metric | Description |
GPU compute utilization (by node) | The GPU compute utilization for the node. |
GPU memory utilization (by node) | The GPU memory utilization for the node. |
Scheduled GPUs (by node) | The number of scheduled GPUs for the node. |
Total GPUs (by node) | The total number of GPUs for the node. |
GPU device power (by node) | The GPU device power consumption for the node. |
Scheduled CPUs (by node) | The number of scheduled CPU cores for the node. |
Total CPUs (by node) | The total number of CPU cores for the node. |
CPU utilization (by node) | The CPU utilization percentage for the node. |
Memory utilization (by node) | The memory utilization percentage for the node. |
For more metrics, see PAI-Quota metrics. | |
View monitoring dashboards
Log on to the PAI console. On a resource quota's details page, click the Monitoring tab to view its monitoring data.
The monitoring page displays metrics for GPU, CPU, memory, network, and disk. You can view these metrics by quota or by node. Note: Monitoring data is retained for up to 30 days.
Click More to select the performance metrics to display. You can also drag the metrics to reorder them based on priority.
On the dashboard, you can zoom in on a selected area, undo a zoom, reset the view, and download chart data.
If you enable chart synchronization, zoom and pan actions are applied to all charts simultaneously, making it easier to compare different views.
You can customize the number of charts displayed per row.
Use CloudMonitor
CloudMonitor is an Alibaba Cloud service that monitors your resources. You can use CloudMonitor to view PAI-Quota monitoring data, set up alerts, and subscribe to metric data through APIs to build custom dashboards. For more information, see What is CloudMonitor?.
Billing
Using CloudMonitor incurs fees. For billing details, see CloudMonitor billing.
View monitoring data
-
Log on to the Cloud Monitor console.
-
In the left-side navigation pane, choose .
On the Cloud Service Monitoring page, select PAI-Quota. In the search box, select or search for a resource quota name. The corresponding monitoring charts are displayed at the bottom of the page.
On the monitoring charts, you can:
Switch monitoring dimensions: You can view metrics by quota or by node.
Switch the time range: You can select 1 hour, 3 hours, 6 hours, 12 hours, 1 day, 3 days, 7 days, 14 days, or a custom time range.
Zoom in: Click the zoom icon
in the upper-right corner of a chart to view its details.
Configure monitoring and alerting
You can use the alerting feature to monitor resource usage within your resource quotas and configure alert rules. If resource usage fluctuates and exceeds a specified threshold, the system sends an alert. To configure alerts, follow these steps:
Step 1: Configure an alert contact
-
Log on to the Cloud Monitor console.
-
In the left-side navigation pane, choose .
On the Alert Contacts tab, click Create Contact, enter the name, phone number, email address, or webhook URL for the contact, and click Confirm.
On the Alert Contact Group tab, click Create Contact Group, enter a name for the group, add existing alert contacts to the group, and then click Confirm.
Step 2: Configure an alert rule
-
In the left-side navigation pane of the CloudMonitor console, choose Cloud Resource Monitoring > Cloud Service Monitoring.
On the Cloud Product Monitoring page, search for and go to PAI-Quota.
On the PAI-Quota page, select the region where your service is located, and then click Create Alert Rule.
In the Create Alert Rule panel, configure the following parameters and click Confirm.
Parameter
Description
Product
The product to which the alert rule applies. Select PAI-Quota.
Resource range
The scope of the alert rule. Valid values: All Resources and Instances.
All Resources: An alert notification is sent if any resource meets the alert rule condition.
Instances: Select the associated resources, which are the resource quota IDs. An alert is triggered only when one or more of the selected instances meet the alert condition.
Rule description
The condition that triggers the alert. For more information about how to configure an alert rule, see Create an alert rule.
Mute for
The interval for resending notifications for an unresolved alert.
Effective period
The time period during which the alert rule is active.
Alert contact group
The contact group that receives alert notifications. Select a group to which alert contacts are assigned.
Tag
A custom tag for the alert rule, which consists of a tag key and a tag value.
On the PAI-Quota page, click View Alert Rules to view the details of created alert rules, view alert history, and modify rules.
You can also configure alerts by using API operations. These operations allow you to view alert history, manage alert templates, and configure alert rules and contacts. For more information, see Alerts.
Subscribe to metrics
CloudMonitor provides an API service that allows you to subscribe to resource quota metrics and data. You can use this service to build your own monitoring systems and dashboards. For more information, see Cloud product monitoring.
|
API |
Description |
|
Queries the latest monitoring data of a specified metric. |
|
|
Queries the monitoring data of a specified metric for a specified cloud service. |
|
|
Queries the monitoring data of a metric for a specified cloud service. |
|
|
Lists available metrics and their metadata. |
|
|
Lists cloud services that provide time-series metrics. |
|
|
Queries the latest monitoring data of a specified metric for a specified cloud service, and then queries the sorted monitoring data of the metric. |
The following example shows how to call the DescribeMetricList operation to query metric data for a specified metric.
Go to the PAI-Quota metrics page.
In the Actions column for the target metric, click Get Metric Data.
In OpenAPI Explorer, configure the following key parameters and use the default settings for the other parameters. For more information about the parameters, see DescribeMetricList.
Parameter
Description
Namespace
Set this to
acs_pai_quota.MetricName
The name of the metric to query. Example:
QUOTA_CPU_REQUEST.StartTime
The start of the time range for the query. Example: 2024-05-15 00:00:00.
EndTime
The end of the time range for the query. Example: 2024-05-28 00:00:00.
NoteThe time range cannot exceed 31 days.
After you configure the parameters, click Initiate Call to view the metric data for the specified time range.
Use ARMS
Application Real-Time Monitoring Service (ARMS) is an Alibaba Cloud-native observability platform. You can use ARMS to create custom Grafana dashboards for PAI-Quota and configure Prometheus alert rules to fully monitor resource quota metrics. For more information, see What is Application Real-Time Monitoring Service (ARMS)?.
Billing
Using ARMS incurs fees. For billing details, see ARMS billing.
Integrate monitoring data
Procedure:
-
Log on to the ARMS console.
In the left-side navigation pane, click Integration Center.
On the Integration Center page, click the AI tab on the left, and then click Alibaba Cloud PAI-Quota.
(Optional) In the panel that appears, you can preview the monitoring dashboard and review the collected metrics and alert rule templates.
Preview
Click the Preview tab to view the metrics dashboard.
Metrics
Click the Metrics tab to view the list of collected metrics.
Alert rule templates
Click the Alert Rule Templates tab to view the predefined alert rule templates.
On the Start Integration tab of the panel, configure the following parameters and click Confirm.
Parameter
Description
Region for data storage
Select the region where you want to store the data.
Integration name
Enter a name for the integration.
The integration of monitoring data takes about one to two minutes.
After the installation is complete, you can click Integration Management to view details of the integrated environment.
View the Grafana dashboard
Log on to the ARMS console. In the left-side navigation pane, choose Integration Management. On the Integrated Environments tab, select Cloud Service Environment, and then click the name of the environment.
On the Component Management tab, in the Component Type section, click Dashboard to view the built-in Grafana dashboard.
Click the dashboard name to view the monitoring dashboard.
Configure Prometheus alerts
You can configure monitoring alerts by using Prometheus. Procedure:
Log on to the ARMS console. In the left-side navigation pane, choose Integration Management. On the Integrated Environments tab, select Cloud Service Environment, and then click the name of the environment.
On the Component Management page, in the Component Type section, click Alert Rule to view the built-in alert rules.
The built-in rules generate alert events but do not send notifications by default. To receive notifications, use one of the following methods:
Configure a notification policy to define matching rules for alert events. When a rule is triggered, the system sends an alert message to the specified recipient by using your chosen method. For more information, see Notification policies.
Edit an alert rule to configure its notification method.
On the page for editing a Prometheus alert rule, you can customize alert conditions, duration, content, and notification methods. For more information, see Create a Prometheus alert rule.