Platform For AI:Monitor training jobs and configure alerts

Billing

CloudMonitor incurs fees. For billing details, see Billing of CloudMonitor.

View monitoring data

1. Log on to the Cloud Monitor console.

2. In the left-side navigation pane, choose Visualization > Cloud Service Monitoring Dashboard.

3. On the Cloud Service Monitoring Dashboard page, select PAI-Deep Learning Containers (DLC) and then select or search for a Workspace ID to view the corresponding monitoring charts. To find a workspace ID, see Manage workspaces.

   

   On the monitoring charts, perform the following operations:

   • Switch monitoring dimensions: The system displays monitoring metrics by job, pod (worker), and GPU dimension.

     • Click the Job dimension tab. Select or enter a DLC job ID to view the monitoring data for a single job.

     • Click the Pod dimension tab. Select or enter a pod ID to view the monitoring data for a single pod.

     • Click the GPU Level tab. Select or enter a pod ID to view GPU-level monitoring data for a single pod.

   • Change the time range:

   • Zoom in: Click the zoom-in icon in the upper-right corner of each chart to view detailed monitoring data.

Configure alerts

Configure alert rules to monitor DLC job resource levels. When a metric violates a rule, the system sends a notification. Configure alerts through the CloudMonitor console or APIs.

Configure alert contacts

1. Log on to the Cloud Monitor console.

2. In the navigation pane on the left, choose Alert Service > Alert Contacts.

3. On the Alert Contacts tab, click Create Alert Contact. Enter the contact's name, phone number, email address, or webhook URL, then click OK.

4. On the Alert Contact Group tab, click Create Alert Contact Group. Enter a group name, add existing alert contacts to the group, then click OK.

Configure alert rules

1. In the left-side navigation pane of the CloudMonitor console, choose Cloud Service Monitoring.

2. On the Cloud Service Monitoring page, search for and select PAI-Deep Learning Containers (DLC).

3. On the PAI-Deep Learning Containers (DLC) page, select the region where your service is located and click Create Alert Rule.

4. In the Create Alert Rule panel, configure the following parameters and click OK.

   Parameter	Description
   Product	Select PAI-Deep Learning Containers (DLC).
   Resource scope	Scope of the alert rule: All Resources or Instance. All Resources: An alert is sent if any DLC resource meets the alert rule. Instance: In the Associate Resources section, add workspaces. An alert is sent only when a DLC job in those workspaces meets the alert rule.
   Rule	Conditions that trigger the alert. For more information, see Create an alert rule.
   Mute for	Interval at which alert notifications are resent if the alert is not resolved.
   Effective period	Time period during which the alert rule is active.
   Tags	Custom tag for the alert rule, consisting of a key-value pair.
   Contact group	Contact group that receives alert notifications.

5. On the PAI-Deep Learning Containers (DLC) page, click Alert Rules to view the details and history of alert rules.

Subscribe to monitoring metrics

CloudMonitor provides APIs for subscribing to DLC monitoring metrics to build custom monitoring systems and dashboards. For the procedure, see Cloud Service Monitoring API directory.

The following example uses the DescribeMetricList API to query monitoring data for a DLC metric.

1. Go to the Metrics for Deep Learning Containers (DLC) page.

2. In the metrics list, find the target metric and click Get Metric Data in the Actions column.

3. On the OpenAPI Explorer page, configure the following key parameters and leave the others at their default values. For more information, see DescribeMetricList.

   Parameter	Description
   Namespace	Set this to acs_pai_dlc.
   MetricName	Target monitoring metric. For example, CARD_GPU_DRAM_ACTIVE_UTIL.
   StartTime	Start time. For example, 2024-05-15 00:00:00.
   EndTime	End time. For example, 2024-05-28 00:00:00. Note The interval between StartTime and EndTime must be 31 days or less.

4. After you configure the parameters, click Initiate Call to view the monitoring data for the specified time range.

Billing

ARMS incurs fees. For billing details, see Billing of ARMS.

Integrate monitoring data

To integrate DLC monitoring data into ARMS:

1. Log on to the ARMS console, and in the left-side navigation pane, click Integration Center.

2. On the Integration Center page, click the Artificial Intelligence tab, then click Alibaba Cloud PAI-DLC Distributed Training Service.

3. In the panel that appears, on the Start Provisioning tab, select a Data Storage Region, enter an Integration Name, and then click OK.

   The integration takes about 1 to 2 minutes. Switch to the Effect Preview, Collected Metrics, and Alert Rule Templates tabs to view the metric dashboard, supported metrics, and alert rule templates.

4. After the integration is complete, go to the Provisioning page to view the integrated environment details.

View Grafana dashboards

1. Log on to the ARMS console. In the left-side navigation pane, select Provisioning. On the Provisioned Environments > Cloud Service Region Environment tab, click the environment name.

2. On the Component Management tab, in the Component Type section, select Alibaba Cloud PAI-DLC Distributed Training Service, and then click Dashboards on the right to view the built-in Grafana dashboards.

3. Click a dashboard name to view the monitoring dashboard.

Configure Prometheus alerts

Configure Prometheus alert rules for DLC training jobs:

1. Log on to the ARMS console. In the left-side navigation pane, select Provisioning. On the Provisioned Environments > Cloud Service Region Environment tab, click the environment name.

2. On the Component Management tab, in the Component Type list, select Alibaba Cloud PAI-DLC Distributed Training Service and click Alert Rules to view the built-in alert rules.

3. The built-in alert rules generate events but do not send notifications. Configure notifications in one of the following ways:

   • Set up a notification policy with matching rules for alert events. When a rule matches, the system sends an alert to the specified recipient. For more information, see Notification policies.

   • Edit the alert rule to configure the notification method. On the Prometheus alert rule editing page, customize alert conditions, duration, content, and notifications. For details, see Create a Prometheus alert rule.