Data Lake Analytics (DLA) provides the performance monitoring feature for virtual clusters in the serverless Spark engine. This topic describes how to view the metrics of virtual clusters in the DLA console.

Prerequisites

  • A virtual cluster of DLA is purchased.
  • The AliyunARMSFullAccess policy is attached to the RAM user that you use. This prerequisite must be met if you want to use the credentials of a RAM user to view the metrics of virtual clusters.

View the metrics related to cluster monitoring

  1. Log on to the DLA console.
  2. In the left-side navigation pane, click Virtual Cluster management.
  3. Find the virtual cluster whose metrics you want to view and click Details in the Actions column. Virtual Cluster management
  4. In the left-side navigation pane, click Monitoring alarm. On the page that appears, click the Standard monitoring tab. Cluster monitoring tab
Note For more information about the metrics on the Cluster monitoring tab, see Metrics.

View the metrics related to job monitoring

You can view the metrics related to job monitoring on the Job list or Job attempt list tab.

  • View the metrics on the Job list tab.
    1. Log on to the DLA console.
    2. In the left-side navigation pane, choose Serverless Spark > Submit job.
    3. On the Job list tab, find the job whose metrics you want to view and select Monitoring from the Operation drop-down list in the Operation column. Job list tabJob monitoring
      Note For more information about the metrics on the Job monitoring tab, see Metrics.
  • View the metrics on the Job attempt list tab.
    1. Log on to the DLA console.
    2. In the left-side navigation pane, choose Serverless Spark > Submit job.
    3. On the Job attempt list tab, find the job whose metrics you want to view and select Monitoring from the Operation drop-down list in the Operation column. Job attempt list tabJob monitoring
      Note For more information about the metrics on the Job monitoring tab, see Metrics.

Metrics

The following table describes the metrics that are displayed on the Cluster monitoring or Job monitoring tab of the Standard Monitoring tab.
Category Metric Description
Cluster monitoring VC CPU Quota The maximum number of CPU cores that can be used in a virtual cluster and the number of CPU cores that are used.
VC Memory Quota The maximum size of memory that can be used in a virtual cluster and the size of the memory that is used.
Job monitoring CPU Usage The CPU utilization of the driver and executors on which the current job is run.
Memory Usage The memory usage of the driver and executors on which the current job is run.
Network I/O The network transmission speed of the driver and executors on which the current job is run.
Minor GC(GC time/1 min) The time spent in a minor garbage collection (GC) per minute on the driver and executors on which the current job is run.
Full GC(GC time/1 min) The time spent in a full GC per minute on the driver and executors on which the current job is run.
Streaming Processing Rate/Min The rate at which the current streaming job processes records. Unit: rows/minute.
Streaming Processing Delay The processing delay of the current streaming job.
Streaming Scheduling Delay The scheduling delay of the current streaming job.
Structured Streaming Latency The delay of the current structured streaming job.
Structured Streaming Processing Rate The rate at which the current structured streaming job processes records. Unit: rows/second.
Structured Streaming Input Rate The rate at which the current structured streaming job reads data. Unit: rows/second.