After auto scaling fires, confirm that nodes were added or removed as expected and diagnose any failures using the Scaling Activity Analysis subtab in the EMR console. The subtab provides six trend charts and a searchable activity log for your E-MapReduce (EMR) cluster.
Prerequisites
Before you begin, ensure that you have:
-
A DataLake cluster, Dataflow cluster, online analytical processing (OLAP) cluster, DataServing cluster, or custom cluster. For more information, see Create a cluster.
-
An auto scaling rule configured for the cluster.
View scaling activity analysis
-
Log on to the EMR console. In the left-side navigation pane, click EMR on ECS.
-
In the top navigation bar, select a region and a resource group.
-
On the EMR on ECS page, click the cluster name in the Cluster ID/Name column.
-
Click the Auto Scaling tab, then click the Scaling Activity Analysis subtab.
Data charts
The subtab displays six charts. Use these charts to get a high-level view of scaling trends before drilling into individual activity records.
| Chart | What it shows |
|---|---|
| Cluster Overview | Hourly changes in the number of running nodes and cores |
| Distribution of Cluster Instances by Billing Method | Hourly changes in the number and core count of pay-as-you-go instances and spot instances |
| Cluster Core-Hour Statistics | Hourly changes in the core count of pay-as-you-go and spot instances |
| Cluster Auto Scaling Overview | Hourly count of nodes created and released by auto scaling |
| Number of Successful/Failed Scaling Activities | Hourly count of successful and failed scaling activities |
| Distribution of Scaling Activity Failure Reasons | Distribution of failure reasons for scaling activities within the selected time period |
Activity records
Below the charts, the activity records table lists each scaling event. Filter records by rule name, node group ID, status, date, or node name.
Records are retained for 60 days. The system automatically removes older entries.
The following table describes each activity status.
| Status | Meaning |
|---|---|
| In Progress | The scaling activity is in progress |
| Execution Succeeded | All nodes specified in the scaling rule were added or removed |
| Partially Successful | Some nodes were added or removed; the remaining nodes failed due to an insufficient disk quota or insufficient Elastic Compute Service (ECS) resources |
| Failed | No nodes were added or removed |
| Rejected | The scaling activity was rejected because the node count in the scaling group has already reached the upper or lower limit, or the previous activity is not yet complete |