Platform For AI:Training monitoring and alerts

Billing

Using the CloudMonitor service incurs fees. For detailed billing information, see Billing of CloudMonitor.

View monitoring data

1. Log on to the Cloud Monitor console.

2. In the left-side navigation pane, choose Visualization > Cloud Service Monitoring Dashboard.

3. On the cloud service monitoring dashboard page, select PAI-Deep Learning Containers (DLC), and then in the search box, select or search for a workspace ID to view the corresponding monitoring charts. To find your workspace ID, see Manage workspaces.

   The monitoring charts area displays three GPU metric panels on the job dimension tab: GPU Memory Interface Utilization (Job Dimension) (%), GPU Compute Utilization (Job Dimension) (%), and GPU SM Utilization (Job Dimension) (%). The reporting period for these metrics is 10 seconds.

   On the monitoring charts, you can:

   • Switch monitoring dimensions: Display metrics by job, pod (worker), or GPU dimension.

     • Click the job dimension tab. Select or enter a DLC job ID to view the monitoring data for a single job.

     • Click the pod dimension tab. Select or enter a pod ID to view the monitoring data for a single pod.

     • Click the GPU Level tab. Select or enter a pod ID to view GPU-dimension monitoring data for a single pod in a specified DLC job.

   • Change the time range: You can select 1 hour, 3 hours, 6 hours, 12 hours, 1 day, 3 days, 7 days, 14 days, or a Custom time period.

   • Zoom in: Click the zoom-in icon in the upper-right corner of each chart to view detailed monitoring data.

Configure alerts

You can configure alert rules to monitor the resource levels of PAI-Deep Learning Containers (DLC) jobs. If a resource metric violates a rule, the system sends an alert notification. This section describes how to configure alerts by using the CloudMonitor console and APIs.

Configure alert contacts

1. Log on to the Cloud Monitor console.

2. In the left-side navigation pane, choose Alerts > Alert Contacts.

3. On the Alert Contacts tab, click Create Contact, enter the name, phone number, email address, or webhook URL for the contact, and click Confirm.

4. On the Alert Contact Group tab, click Create Contact Group, enter a name for the group, add existing alert contacts to the group, and then click Confirm.

Configure alert rules

1. In the left-side navigation pane of the CloudMonitor console, choose Cloud Resource Monitoring > Cloud Service Monitoring.

2. On the Cloud Service Monitoring page, open PAI-Deep Learning Containers (DLC). In the search box, enter PAI-Deep Learning Containers (DLC). In the search results, under the Metric Monitoring category, click PAI-Deep Learning Containers (DLC).

3. On the PAI-Deep Learning Containers (DLC) page, select your service region and click Create Alert Rule.

4. In the Create Alert Rule panel, configure the following parameters and click Confirm.

   Parameter	Description
   Product	The product to monitor. Select PAI-Deep Learning Containers (DLC).
   Resource Scope	The scope of the alert rule. Options: All Resources and instance. All Resources: An alert notification is sent if any DLC resource meets the alert rule. instance: In the Associate Resources section, you must add the workspaces that you want to associate. An alert notification is sent only when a DLC job in the added workspaces meets the alert rule.
   Rule Description	The conditions that trigger the alert. An alert is triggered when monitoring data meets these conditions. For more information about how to set a rule description, see Create an alert rule.
   Mute For	The resend interval for unresolved alerts.
   Effective Period	The period when the alert rule is active. CloudMonitor only checks for alerts during this period.
   Tags	Custom tags for the alert rule, specified as key-value pairs.
   Alert Contact Group	The contact group that receives alert notifications. Select a group with configured alert contacts.

5. On the PAI-Deep Learning Containers (DLC) page, click View Alert Rules to view the details and history of your alert rules. You can also modify the rules.

Subscribe to monitoring metrics

CloudMonitor provides a comprehensive set of API operations that you can use to subscribe to DLC monitoring metrics and data. This allows you to build your own monitoring systems and dashboards. For detailed steps, see Cloud Service Monitoring API Directory.

The following example shows how to call the DescribeMetricList API operation to query the monitoring data of a specified metric for PAI-Deep Learning Containers (DLC).

1. Go to the Metrics for PAI-Deep Learning Containers (DLC) page.

2. On the metrics list page, find the target metric, such as JOB_GPU_ACCELERATOR_DUTTY_UTIL, and click Get Metric Data in the Actions column to go to the OpenAPI Portal page.

3. On the OpenAPI Portal page, configure the following key parameters and leave the others at their default values. For more information about the parameters, see DescribeMetricList.

   Parameter	Description
   Namespace	Set this parameter to acs_pai_dlc.
   MetricName	The metric to query. Example: CARD_GPU_DRAM_ACTIVE_UTIL.
   StartTime	The start time. Example: 2024-05-15 00:00:00.
   EndTime	The end time. Example: 2024-05-28 00:00:00. Note The interval between StartTime and EndTime must be 31 days or less.

4. After you configure the parameters, click Initiate Call to view the monitoring data for the specified time range. A successful call returns a 200 status code. The Datapoints array in the response body contains data fields such as timestamp, jobId, regionId, userId, workspaceId, and Value.

Billing

Using ARMS incurs fees. For billing details, see ARMS billing.

Integrate monitoring data

To integrate DLC monitoring data into ARMS:

1. Log on to the ARMS console, and in the left-side navigation pane, click Integration Center.

2. On the Integration Center page, click the Artificial Intelligence tab, and then click PAI-DLC Distributed Training Service.

3. In the panel that appears, on the Start Provisioning tab, select a Data Storage Region, enter an Integration Name, and then click OK.

   The integration takes about 1 to 2 minutes. You can also switch to the Effect Preview, Collected Metrics, and Alert Rule Templates tabs to view the metric dashboard, supported metrics, and alert rule names and template details.

4. After the integration is complete, you can click Provisioning to view the details of the provisioned environment.

Grafana dashboard

1. Log on to the ARMS console. In the left-side navigation pane, select Provisioning. On the Provisioned Environments > Cloud Service Region Environment tab, click the environment name.

2. On the Component Management tab, in the Component Type section, select PAI-DLC Distributed Training Service, and then click Dashboards on the right to view the built-in Grafana dashboards.

3. Click a dashboard name to view the monitoring dashboard. The PAI-DLC Distributed Training Service - Instance Details dashboard provides filters for workspaceId, jobId, pod, and gpu. It organizes metrics into Job Dimension, Card Dimension, and Pod Dimension panels. The Pod Dimension panel shows a pod details table, including CPU utilization, disk I/O rates, and memory usage, along with time-series charts for CPFS read latency, CPFS write data volume, and CPFS read data volume.

Configure Prometheus alerts

You can configure monitoring alerts using Prometheus as follows:

1. Log on to the ARMS console. In the left-side navigation pane, select Provisioning. On the Provisioned Environments > Cloud Service Region Environment tab, click the environment name.

2. On the Component Management tab, in the Component Type list, select PAI-DLC Distributed Training Service and click Alert Rules to view the built-in alert rules. By default, these rules are in the Stopped state.

3. The built-in alert rules generate events but do not send notifications. You can configure notifications to be sent to an email address or other platforms in one of the following two ways:

   • Set up a notification policy. This policy defines matching rules for alert events. When an alert event matches a rule, the system sends a notification to the specified recipient through your chosen notification method. For more information, see Notification Policies.

   • Edit the alert rule to configure the notification method. On the alert rule management page, select the target component type from the left-side navigation pane, such as PAI-DLC Distributed Training Service, PAI-DSW, PAI-Quota Service, or PAI-Quota (non-Lingjun). Find the target rule in the list and click Edit. On the Prometheus alert rule editing page, you can customize the alert conditions, duration, content, and notifications. For more information, see Create a Prometheus alert rule. When editing the rule, set Check Type to Custom PromQL. In the Custom PromQL Statement field, enter an expression, such as AliyunPaidlc_POD_STATE_ACTIVE{} > 80. Set the Duration to 2 minutes and the Alert Level to P2. In the Alert Notification section, select Simple Mode and configure the Recipient, Notification Period (from 00:00 to 23:59), and Repeat Policy. After you complete the configuration, click Done.