EMR's YARN dashboard lets you monitor cluster resource utilization and evaluate auto scaling effectiveness. Use it to spot over-provisioned capacity, review 30-day utilization trends, and tune auto scaling rules based on historical YARN metrics.
Prerequisites
Before you begin, ensure that you have:
-
A DataLake cluster or a custom cluster with the YARN service enabled. For setup instructions, see Create a cluster.
Methods at a glance
| Goal | Method |
|---|---|
| Compare utilization trends and get a recommended scaling rule | View the YARN resource utilization trend |
| Inspect YARN metric data and evaluate auto scaling effect | View YARN metric data |
View the YARN resource utilization trend
Use this method to compare resource utilization over time and check whether your cluster is over-provisioned. The console surfaces a recommended scaling rule based on the trend data.
-
Log in to the EMR console. In the left-side navigation pane, click EMR on ECS.
-
In the top navigation bar, select the region where your cluster resides and select a resource group.
-
On the EMR on ECS page, click the cluster name in the Cluster ID/Name column.
-
Click the Auto Scaling tab, then open the Configure Auto Scaling subtab.
-
Review the resource utilization trend line, which shows changes over the previous 30 days. To check for resource waste, compare it against the trend from the previous N days, where N is an integer less than 30. The console displays a recommended auto scaling rule based on this analysis. Configure scaling for the cluster node group based on the recommended rule.
View YARN metric data
Use this method to inspect raw YARN metrics, observe container and node utilization, and evaluate the effect of auto scaling on your cluster.
-
Log in to the EMR console. In the left-side navigation pane, click EMR on ECS.
-
In the top navigation bar, select the region where your cluster resides and select a resource group.
-
On the EMR on ECS page, click the cluster name in the Cluster ID/Name column.
-
Click the Monitoring and Diagnostics tab, then click the Metric Monitoring subtab.
-
In the Dashboard drop-down list, select YARN-HOME. The dashboard displays YARN metric data for the cluster.
-
You can select a time range to view resource utilization over the specified period.
-
Locate the Yarn Scaling metric in the dashboard. Yarn Scaling is a metric used to show YARN auto scaling. It shows the resource utilization of containers and nodes running in your cluster, helping you understand the auto scaling effect of the cluster and accurately evaluate its performance.