This topic provides an overview of the monitoring for the Spark service.

Prerequisites

A Hadoop cluster is created. For more information, see Create a cluster.

Go to the SPARK Monitoring page

  1. Log on to the Alibaba Cloud EMR console.
  2. In the top navigation bar, select the region where your cluster resides and select a resource group based on your business requirements.
  3. Click the Monitor tab.
  4. In the left-side navigation pane, click Cluster Monitoring.
  5. On the Cluster Status page, find the cluster whose monitoring data you want to view, and click Details in the Action column.
  6. In the left-side navigation pane, choose Service Monitoring > Spark.

    The SPARK Monitoring page appears.

SPARK Monitoring page

The SPARK Monitoring page displays basic JVM metric charts, recent alerts and exceptions on the current day, and start or stop history of the SparkHistory process for the Spark service.

  • Alerts and Details: This section displays critical exception events related to the Spark service on the current day.
  • SparkHistory Process JVM Indicators: This section includes SparkHistory Process Memory Usage, SparkHistory Process Garbage Collection Time, SparkHistory Process Garbage Collections, Heap Memory, and Non-Heap Memory.
  • SparkHistory Process File Descriptor: This section displays the maximum number of file descriptors that the SparkHistory process can use and the number of file descriptors that are in use.
  • SparkHistory Process History.
    Parameter Description
    Date The time when an operation is performed.
    Start/Restart/Stop The operation type, which can be start, stop, or restart.
    Auto Resume Indicates whether an operation is automatically resumed by the keepalive mechanism of EMR. The EMR agent automatically resumes components that exit abnormally to ensure service availability.
    Started By The Linux user who performs an operation. This parameter is left empty for a process in the Stop state.
    PID The ID of a process generated in an operation. This parameter is left empty for a process in the Stop state.
    PPID The ID of a parent process generated in an operation. This parameter is left empty for a process in the Stop state.
    Startup Parameters The detailed start parameters for a process generated in an operation. This parameter is left empty for a process in the Stop state.