E-MapReduce:Configure auto scaling rules for a cluster

Time-based scaling

Configure a time-based scale-out rule that runs repeatedly or only once, based on when your workload increases. You can also configure a scale-in rule to reduce nodes during off-peak hours. For recurring rules, set the Rule Expiration Time parameter to specify when the rule expires. After expiration, no scaling activity is triggered.

For example, if your workloads increase at 22:00 and decrease at 04:00 every day, configure a recurring scale-out rule at 22:00 and a recurring scale-in rule at 04:00.

For more information about the parameters and cluster load metrics, see Create custom auto scaling policies.

Load-based scaling

EMR recommends common metrics for scaling by default. Configure specific thresholds based on your cluster's metric trends. After you set the values, click OK and then click Save and Apply. The rules are then automatically triggered in response to workload changes.

Complete the following steps to configure scaling rules for your cluster.

1. Select appropriate metrics.

   On the Monitoring tab, go to the Metric Monitoring page. From the Dashboard drop-down list, select YARN-HOME. Observe how metrics have changed with your workload over time to select the most appropriate ones.

   The metric values must be inversely related to capacity changes — after a scaling activity, the metric values should decrease as the number of instances changes.

   For example, you can configure a scale-out rule that adds one node if the average value of yarn_resourcemanager_queue_AppsPending is greater than or equal to 1 for one consecutive occurrence. After this scale-out activity, the number of pending applications in the queue should decrease.

   The following table lists the recommended metrics.

   Metric	Service	Description
   yarn_resourcemanager_queue_AvailableMBPercentage	YARN	The percentage of available memory resources in the root queue.
   yarn_resourcemanager_queue_AvailableVCores	YARN	The number of available vCPUs that can be allocated in the root queue.
   yarn_resourcemanager_queue_AvailableMB	YARN	The amount of available memory that can be allocated in the root queue. Unit: MB.
   yarn_resourcemanager_queue_AppsPending	YARN	The number of pending applications in the root queue.
   yarn_resourcemanager_queue_PendingContainers	YARN	The number of containers pending allocation in the root queue.
   yarn_resourcemanager_queue_AvailableVCoresPercentage	YARN	The percentage of available vCPU resources in the root queue.

2. Configure an appropriate scaling rule.

   • As a starting point, use pending-related metrics for scale-out rules and available-related metrics for scale-in rules.

   • Within a single rule, define multiple metric-based conditions with AND/OR logic for more granular control over scaling triggers.

   • To prevent resource waste from frequent scaling, set a cooldown time for your scale-in rules. During the cooldown time, the system does not trigger new scaling activities, even if the conditions are met.

     Adding nodes takes an average of 1.55 minutes, and adding 100 nodes takes only about 1.83 minutes. Set the cooldown time for a scale-out rule to 100–300 seconds. This allows the new nodes to take effect before evaluating whether the cluster load metrics have decreased and whether another scale-out is needed.

   • To respond more quickly to metric changes, set the Statistical Period to 1 minute. A longer statistical period may trigger scaling based on outdated data, leading to unnecessary resource usage.

   • Configure the number of instances to add or remove based on the processing capacity of your current nodes and your expected workload growth. Estimate the number of nodes required to bring the metric back to the desired range.

   • Set time constraints (the time range within a day when the rule is active) to combine different scaling rules and apply different scaling conditions during different time periods.

3. Set the minimum and maximum size.

   The Limits on Node Quantity of Current Node Group parameter limits the number of nodes in the current node group. The Maximum Number of Instances parameter sets the upper limit, preventing the node group from scaling out indefinitely. The Minimum Number of Instances parameter sets the lower limit to ensure enough nodes are available for your workload. If instances are unexpectedly released, the system adds instances to maintain the minimum count.

4. Adjust the rules.

   After configuring the rules, monitor your metrics and scaling activity history to adjust parameters as needed.

   • If scaling activities are too frequent and newly added instances are idle or quickly scaled in, consider making your trigger conditions more restrictive by using the AND operator, or extend the rule's Cooldown Time.

   • If multiple scale-out events are required to process a single job, or if scaling is not fast enough to keep up with your workload, consider increasing the number of instances added in each scaling activity.