E-MapReduce:Configure and view monitoring and alerts for Spark jobs

Step 1: Enable PodMonitor

1. Log on to the ARMS console.

2. In the left-side navigation pane, click Integration Management.

3. On the Integrated Environments tab, click the Container Service tab, find your ACK environment instance, and click Metric Scraping in the Actions column.

4. On the Metric Scraping tab, click Pod Monitor.

5. On the Pod Monitor tab, find each of the following monitors and click Enable in the Actions column:

   • sparkoperator-podmonitor

   • sparkoperator-spark-podmonitor

   • shuffleservice-master-podmonitor

   • shuffleservice-worker-podmonitor

   If no Shuffle Service cluster is associated with your Spark cluster, skip shuffleservice-master-podmonitor and shuffleservice-worker-podmonitor.

Step 2: Submit a Spark job

Submit a Spark job. See Submit a Spark job.

Step 3: Verify metrics on the Grafana dashboard (optional)

After submitting a job, confirm that metrics are being collected by querying them in Grafana.

1. Log on to Grafana.

2. In the left-side navigation pane, click the Explore icon.

3. On the Explore page, select your ACK cluster from the drop-down list, specify a metric name, and click Run Query in the upper-right corner. Use the following table to identify the metric prefix for what you want to monitor:

   What you want to monitor	Metric prefix
   Spark pod metrics (driver, executor, or JVM)	spark_driver_, spark_executor_, or jvm_
   Spark application-level metrics (via Spark Operator)	spark_app
   Shuffle Service metrics	metrics_

Step 1: Go to the Alert Rules page

1. Log on to the EMR console. In the left-side navigation pane, click EMR on ACK.

2. On the EMR on ACK page, find your cluster and click the link in the ACK Cluster column.

3. In the left-side navigation pane, choose Operations > Prometheus Monitoring.

4. On the Prometheus Monitoring page, wait for the system to automatically install the required component and load the dashboards. After installation completes, click each tab to review the corresponding metrics.

5. Click Go to ARMS Prometheus in the upper-right corner.

6. In the left-side navigation pane, click Alerts Rules.

Step 2: Create a Prometheus alert rule

1. On the Prometheus Alert Rules page, click Create Prometheus Alert Rule.

2. On the Create Prometheus Alert Rule page, configure the alert parameters.

Step 3: View alert event history

On the Prometheus Alert Rules page, find the alert rule you want to review and click Alert Event History in the Actions column.

When the configured alert conditions are met, alerts are triggered and listed on this page.