This topic provides an overview of the monitoring for the Spark service.

Prerequisites

A Hadoop cluster is created.

Enter the SPARK Monitoring page

  1. Log on to the Alibaba Cloud E-MapReduce console.
  2. Click the Monitor tab.
  3. In the left-side navigation pane, click Cluster Monitoring.
  4. On the Cluster Status page, click Details in the Action column that corresponds to the Hadoop cluster.
  5. In the left-side navigation pane, choose Service Monitoring > Spark to enter the SPARK Monitoring page.

SPARK Monitoring page

The SPARK Monitoring page displays basic JVM metric charts, recent alerts and exceptions on the current day, and start or stop history of the SparkHistory process for the Spark service.

  • Alerts and Details. This section displays critical exception events related to the Spark service on the current day.
  • SparkHistory Process JVM Indicators. This section includes SparkHistory Process Memory Usage, SparkHistory Process Garbage Collection Time, SparkHistory Process Garbage Collections, Heap Memory, and Non-Heap Memory.
  • SparkHistory Process File Descriptor. This section displays the maximum number of file descriptors that the SparkHistory process can use and the number of file descriptors that are in use.
  • SparkHistory Process History.
    Parameter Description
    Date The time when an operation is performed.
    Start/Restart/Stop The operation type, which can be start, stop, or restart.
    Auto Resume Whether an operation is automatically resumed by the keepalive mechanism of EMR. The EMR agent automatically resumes components that exit abnormally to ensure service availability.
    Started By The Linux user who performs an operation. This parameter is left empty for a process in the Stop state.
    PID The ID of a process generated in an operation. This parameter is left empty for a process in the Stop state.
    PPID The ID of a parent process generated in an operation. This parameter is left empty for a process in the Stop state.
    Startup Parameters The detailed start parameters for a process generated in an operation. This parameter is left empty for a process in the Stop state.