E-MapReduce:Configure auto scaling rules for a cluster

Time-based scaling

You can configure a time-based scale-out rule that is repeatedly executed or executed only once based on the point in time when your business volume is likely to increase. You can also configure a scale-in rule to reduce the number of nodes during off-peak hours. If the time-based scaling rule that you configured is repeatedly executed, you can configure the Rule Expiration Time parameter to specify the expiration time of the rule. After the time-based scaling rule expires, no scaling activity is triggered.

For example, your business workloads increase at 22:00 and decrease at 04:00 every day. In this case, you can configure a time-based scale-out rule that is repeatedly executed at 22:00 every day and a time-based scale-in rule that is repeatedly executed at 04:00 every day.

For more information about the parameters and cluster load metrics, see Create custom auto scaling policies.

Load-based scaling

By default, EMR recommends common metrics for scaling. You must configure specific thresholds based on your cluster's metric trends. After you set the values, click OK and then click Save and Apply. The rules are automatically triggered in response to workload changes.

Follow these steps to configure scaling rules that fit your cluster's needs.

1. Select appropriate metrics.

   On the Monitoring tab, go to the Metric Monitoring page. From the Dashboard drop-down list, select YARN-HOME. Observe how metrics have changed with your workload over time to select the most appropriate ones.

   The values of the metrics must be inversely related to the capacity change. After a scaling activity occurs, the values of the metrics decrease when the number of instances changes.

   For example, you can configure a scale-out rule that adds one node if the average value of yarn_resourcemanager_queue_AppsPending is greater than or equal to 1 for one consecutive occurrence. After this scale-out activity, the number of pending applications in the queue should decrease.

   The following table lists the recommended metrics.

   Metric	Service	Description
   yarn_resourcemanager_queue_AvailableMBPercentage	YARN	The percentage of available memory resources in the root queue.
   yarn_resourcemanager_queue_AvailableVCores	YARN	The number of available vCPUs that can be allocated in the root queue.
   yarn_resourcemanager_queue_AvailableMB	YARN	The amount of available memory that can be allocated in the root queue. Unit: MB.
   yarn_resourcemanager_queue_AppsPending	YARN	The number of pending applications in the root queue.
   yarn_resourcemanager_queue_PendingContainers	YARN	The number of containers pending allocation in the root queue.
   yarn_resourcemanager_queue_AvailableVCoresPercentage	YARN	The percentage of available vCPU resources in the root queue.

2. Configure an appropriate scaling rule.

   • When configuring rules for the first time, a good starting point is to use pending-related metrics for scale-out rules and available-related metrics for scale-in rules.

   • Within a single rule, you can define multiple metric-based conditions and specify AND/OR logic between them for more granular control over scaling triggers.

   • To prevent resource waste from frequent scaling, set a cooldown time for your scale-in rules. During the cooldown time, the system does not trigger new scaling activities, even if the conditions are met.

     The average time period for adding nodes requires 1.55 minutes, and the average time period for adding 100 nodes requires only 1.83 minutes. You can set the cooldown time for a scale-out rule to a value that ranges from 100 to 300 (Unit: seconds). This way, after the new nodes are used, you can check whether the values of the configured cluster load metrics decrease and determine whether another scale-out activity is required. This helps prevent resource waste.

   • To respond more quickly to metric changes, we recommend that you set the Statistical Period to 1 minute. A statistical period that is too large can trigger scaling activities based on outdated historical data, which leads to unnecessary waste.

   • Configure the number of instances to add or remove based on the processing capacity of your current nodes and your expected workload growth. Estimate the number of nodes required to bring the metric back to the desired range.

   • Set time constraints (the time range within a day when the rule is active) to combine different scaling rules and apply different scaling conditions during different time periods.

3. Set the minimum and maximum size.

   The Limits on Node Quantity of Current Node Group parameter specifies the limits on the number of nodes in the current node group. The Maximum Number of Instances parameter specifies the upper limit for the number of nodes in the current node group. This prevents your node group from being indefinitely scaled out. The Minimum Number of Instances parameter specifies the lower limit for the number of nodes that are required to process your business. If your instances are released due to unexpected factors, the system adds instances to meet the minimum number of instances.

4. Adjust the rules.

   After configuring the rules, monitor your metrics and scaling activity history over a period of time to adjust the parameters as needed.

   • If scaling activities are too frequent and newly added instances are idle or quickly scaled in, consider making your trigger conditions more restrictive by using the AND operator, or extend the rule's Cooldown Time.

   • If multiple scale-out events are required to process a single job, or if scaling is not fast enough to keep up with your workload, consider increasing the number of instances added in each scaling activity.