E-MapReduce:Configure custom auto scaling rules

Method 1: Add custom scaling rules for an existing cluster

1. Go to the Auto Scaling tab.

   1. Log on to the EMR console.

   2. In the top navigation bar, select a region and a resource group based on your business requirements.

   3. On the EMR on ECS page, find the desired cluster and click the name of the cluster in the Cluster ID/Name column.

   4. On the page that appears, click the Auto Scaling tab.

2. Configure auto scaling rules.

   1. In the Configure Auto Scaling tab of the Configure Auto Scaling Rule section, click Custom Auto Scaling Rule. In the Modify Auto Scaling Rule message, click Reconfigure. Find the desired node group and click Edit in the Actions column.

   2. In the Configure Auto Scaling panel, configure the following parameters:

      • Limits On Node Quantity Of Current Node Group: specifies the range of the number of nodes in the current node group. This parameter defines the upper and lower limits for the number of nodes in the node group when the node group is automatically scaled in or out based on the auto scaling rules. To change the maximum and minimum numbers of nodes in a node group, click Modify Limit.

        • Maximum Number Of Instances: specifies the upper limit for the number of nodes in the current node group. The node group can no longer be scaled out when the number of nodes in the node group reaches the upper limit.

        • Minimum Number Of Instances: specifies the lower limit for the number of nodes in the current node group. The node group can no longer be scaled in when the number of nodes in the node group reaches the lower limit.

      • Trigger Rule: displays the auto scaling rules that are configured for the current node group. You can also add more auto scaling rules for the node group.

        • Time-based scaling

          If the computing workloads of a cluster fluctuate on a regular basis, you can add or remove a specific number of task nodes at fixed points in time every day, every week, or every month to supplement or save computing resources. This ensures that jobs can be completed at low costs. Auto scaling rules are divided into scale-out rules and scale-in rules.

          Important

          To prevent potential conflicts and ensure that scaling operations proceed smoothly, do not set the execution time of scale-out and scale-in rules to the same point in time.

          The following table describes the parameters for a scale-out rule.

          Parameter	Description
          Scale Out Type	Scale Out by Time.
          Rule Name	The name of the scale-out rule. The names of auto scaling rules must be unique in a cluster.
          Frequency	Execute Repeatedly: Auto scaling is performed at a specific point in time every day, every week, or every month. Execute Only Once: Auto scaling is performed only once at a specific point in time.
          Execution Time	The specific point in time at which the scale-out rule is executed.
          Rule Expiration Time	The expiration time of the scale-out rule. After the scale-out rule expires, no scale-out activity is triggered. This parameter is available only if you set the Frequency parameter to Execute Repeatedly.
          Retry Time Range	The retry interval. Auto scaling may fail at the specified point in time due to various reasons. If the retry interval is specified, the system retries auto scaling every 30 seconds during the retry interval specified by this parameter until auto scaling is performed. Valid values: 0 to 3600. Unit: seconds. For example, auto scaling operation A needs to be performed in a specified time period. If auto scaling operation B is still in progress or is in the cooldown state during this period, operation A cannot be performed. In this case, the system tries to perform operation A every 30 seconds within the retry interval you specified. If required conditions are met, the cluster immediately performs auto scaling.
          Nodes For Each Scale Out	The number of nodes that are added to the node group each time the scale-out rule is triggered.
          Best-effort Delivery	We recommend that you turn on the switch. After you turn on this switch, the system will deliver all successfully created nodes in a best-effort manner to ensure continuity of your business. For example, you configure a scale-out rule to add 100 nodes at 00:00 every day, and only 90 nodes are added. If you do not turn on the Best-effort Delivery switch, the scale-out fails. If you turn on the switch, 90 nodes are delivered.

        • Load-based scaling

          Note

          Only clusters in which the YARN service is deployed support load-based scaling rules.

          You can add load-based scaling rules for task node groups in a cluster for which you cannot accurately estimate the fluctuation of big data computing workloads. Auto scaling rules are divided into scale-out rules and scale-in rules. The following table describes the parameters for a scale-out rule.

          Parameter	Description
          Scale Out Type	Scale Out by Load.
          Rule Name	The name of the scale-out rule. The names of auto scaling rules must be unique in a cluster.
          Load Metric-based Trigger Conditions	The conditions that must be met to trigger the scale-out rule. You must select at least one system-defined load metric. To select multiple system-defined load metrics, click Add Metric. You must configure the following parameters: Load metrics: Select system-defined YARN load metrics. For information about the mappings between EMR auto scaling metrics and YARN load metrics, see Description of EMR auto scaling metrics that match YARN load metrics. Note System-defined load metrics vary based on the cluster type. You can view the supported system-defined load metrics in the EMR console. Statistical method: the rule that is used to specify the threshold for each load metric. The scale-out rule is triggered once if the value of the specified aggregation dimension (average, maximum, or minimum) for each selected load metric reaches the threshold within a statistical period.
          Multi-metric Relationship	The relationship among the metrics that are used to trigger the scale-out rule. Valid values: All Metrics Meet the Conditions and Any Metric Meets the Condition.
          Statistical Period	The statistical period for measuring whether the value of the specified aggregation dimension (average, maximum, or minimum) for each selected load metric reaches the threshold. Data is collected, summarized, and compared based on the specified statistical period. The shorter the statistical period is, the more frequently the scale-out rule is likely to be triggered. Specify the statistical period based on your business requirements.
          Repetitions That Trigger Scale Out	Based on the aggregation method, Auto Scaling records each occurrence when a load metric value exceeds the threshold. It triggers the corresponding scaling rule only after reaching a specified number of occurrences.
          Nodes For Each Scale Out	The number of nodes that are added to the node group each time the scale-out rule is triggered.
          Best-effort Delivery	After you turn on this switch, the system will deliver all successfully created nodes in a best-effort manner. Load-based scaling is affected by metrics. You can determine whether to turn on the switch based on your business requirements.
          Cooldown Time	The interval between two scale-out activities. During the cooldown time, the scale-out rule is not triggered even if the scale-out conditions are met. A scale-out activity is performed after the cooldown time ends and the scale-out conditions are met again. After the node group is scaled out and reaches the expected state, the cooldown time keeps the cluster load metric that triggers subsequent scale-out activities in a stable state.
          Effective Time Period	The time range in which the scale-out rule takes effect. This parameter is optional. By default, the scale-out rule is available 24 hours a day. A scale-out activity is triggered only within the time range specified by this parameter.

   3. Click Save And Apply.

      Scale-out activities are triggered for the node group when the scale-out conditions are met.

Method 2: Add custom auto scaling rules when you create a cluster

1. Log on to the EMR console.

2. In the top navigation bar, select a region and a resource group based on your business requirements.

3. Click Create Cluster. For more information, see Create a cluster.

   Note

   You must add a pay-as-you-go task node group before you can configure custom auto scaling rules.

4. Configure the Cluster Scaling parameter.

   1. Click Custom Auto Scaling Rule. Find the desired node group and click Edit in the Actions column.

   2. In the Configure Auto Scaling panel, configure the parameters. For more information, see Configure auto scaling rules.

   3. Click Save And Apply.

5. Confirm the order.

   After the cluster is created, scaling activities are triggered for the node group when the specific conditions are met.

Method 3: Use SDK to add custom auto scaling rules

When you call a specific API operation to create a cluster or create a task node group, you can configure custom auto scaling rules for the task node group to allow the system to automatically manage nodes. For more information, see CreateCluster and CreateNodeGroup. You can also configure custom auto scaling rules for existing task node groups. For more information, see PutAutoScalingPolicy.

The following Java code shows how to configure a load-based scale-out rule.

// This file is auto-generated, don't edit it. Thanks.
package com.aliyun.sample;

import com.aliyun.tea.*;

public class Sample {

    /**
     * description