EMR provides a YARN-based metric dashboard to view cluster resource utilization and evaluate auto scaling effectiveness. You can also use historical YARN metrics to configure and optimize custom auto scaling rules.
Prerequisites
You have created a DataLake or Custom cluster and selected the YARN service. For more information, see Create a cluster.
Procedure
Method 1: View YARN resource utilization comparison
-
Go to the Auto Scaling page.
-
Log on to the EMR on ECS console.
-
In the top navigation bar, select a region and resource group based on your requirements.
-
Click the name of the target cluster.
-
On the page that appears, click the Auto Scaling tab.
-
-
The Configure Auto Scaling page displays a comparison of YARN resource utilization.
View the resource utilization trend for the last 30 days and compare it with previous days to identify resource waste. The system analyzes current resource status and recommends auto scaling rules. Use these recommendations to configure auto scaling for your cluster node groups. For more information, see Create a custom auto scaling policy and Create a managed auto scaling policy.
Method 2: View detailed YARN metric data
-
Go to the Metric Monitoring page.
-
Log on to the EMR on ECS console.
-
In the top menu bar, select a region and a resource group.
-
Click the name of the target cluster.
-
Click Monitoring and Diagnostics at the top of the page.
-
Click the Metric Monitoring tab.
-
-
From the Dashboard drop-down list, select YARN-HOME. The dashboard displays YARN metric data for the cluster. Adjust the time range to view historical cluster utilization.
-
The YARN Scaling section provides YARN auto scaling metrics. Use these metrics to drill down into container and node resource utilization and evaluate auto scaling effectiveness.