E-MapReduce:Create a custom auto scaling policy

Method 1: Configure rules for an existing cluster

1. Go to the Auto Scaling page.

   1. Log on to the EMR on ECS console.

   2. In the top navigation bar, select a region and resource group based on your requirements.

   3. Click the name of the target cluster.

   4. On the page that appears, click the Auto Scaling tab.

2. Configure scaling rules.

   1. On the Configure Auto Scaling tab, in the Configure Auto Scaling Rule section, click Custom Auto Scaling Rule, select Reconfigure, and then click Edit in the Actions column for the target node group.

   2. In the Configure Auto Scaling panel, configure the parameters.

      • Limits on Node Quantity of Current Node Group: Controls the number of nodes in the group to prevent the number of nodes from becoming too large or too small during auto scaling. To change this configuration, click Modify Limit.

        • Maximum Number of Instances: The maximum number of nodes in the current node group. When this limit is reached, the node group stops scaling out.

        • Minimum Number of Instances: The minimum number of nodes in the current node group. When this limit is reached, the node group stops scaling in.

      • Trigger Rule: Displays and allows you to configure the scaling rules for the current node group.

        • Time-based scaling

          If your cluster's computing workload has predictable peaks and troughs, you can set a fixed time on a daily, weekly, or monthly basis to scale out a specific number of task nodes. This adds computing resources when needed, helping to ensure job completion while saving costs. Scaling rules are divided into scale-out rules and scale-in rules.

          Important

          To prevent conflicts and ensure smooth scaling, do not set the execution times for scale-out and scale-in rules to be identical.

          This section uses a scale-out rule as an example.

          Parameter	Description
          Scale-out type	Scale out by time.
          Rule Name	Scaling rule names must be unique within a cluster.
          Frequency	Execute Repeatedly: The scaling action runs at a specific time daily, weekly, or monthly. Execute Only Once: The scaling action runs once at a specified time.
          Execution Time	The time at which the rule is executed.
          Rule Expiration Time	If you set Frequency to Execute Repeatedly, you can set an expiration time for the rule. The rule stops triggering scaling activities after this time.
          Retry Time Range	A scaling action might not run at its scheduled time for various reasons. If you set a retry time range, the system retries the action every 30 seconds within this range until it succeeds. The value can be from 0 to 3,600 seconds. For example, if action A is scheduled but cannot run because another action B is in progress or in a cooldown period, the system will retry action A every 30 seconds during the specified retry time range. The action runs as soon as conditions permit.
          Nodes for each scale-out	The number of nodes to add when the rule is triggered.
          Best-effort Delivery	We recommend that you enable this feature. After you enable best-effort delivery, the system delivers the maximum possible number of nodes to ensure smooth business operations. For example, you configure the system to scale out by 100 nodes at 00:00 every day. If only 90 nodes are in stock, the process would normally fail. With this feature enabled, the 90 available nodes are delivered.

        • Load-based scaling

          Note

          This feature is supported only when the YARN service is deployed in the cluster.

          If you cannot accurately predict the peaks and troughs of your big data computing workload but have a good understanding of your business model, you can use load-based scaling. Scaling rules are divided into scale-out and scale-in rules. This section uses a scale-out rule as an example.

          Parameter	Description
          Scale-out type	Scale out by load.
          Rule Name	Scaling rule names must be unique within a cluster.
          Load Metric-based Trigger Conditions	The conditions that trigger the load-based scaling rule. You need to select one or more system-defined load metrics. To select multiple metrics, click Add Metric. The following parameters are included: Load metric: The name of a system-supported load metric. For more information about the mapping between E-MapReduce auto scaling metrics and YARN service metrics, see Mapping between E-MapReduce auto scaling metrics and YARN service metrics. Note The supported load metrics vary based on the cluster type. The actual metrics displayed in the console prevail. Aggregation method: The method (average, maximum, or minimum) used to evaluate the load metric against a threshold over a statistical period. The rule is triggered if the aggregated value meets the threshold.
          Multi-metric Relationship	You can select All Metrics Meet the Conditions or Any Metric Meets the Condition.
          Statistical Period	The time window for metric collection and aggregation. A shorter period makes the rule more sensitive to load changes. Select a reasonable statistical period based on your business needs.
          Condition Repetition Threshold	The number of consecutive statistical periods the metric must exceed the threshold before the rule is triggered. This prevents scaling due to brief spikes.
          Nodes for each scale-out	The number of nodes to add when the rule is triggered.
          Best-effort Delivery	If you enable this feature, the system delivers the maximum possible number of nodes. Evaluate whether to enable this feature for load-based scaling. Unlike time-based scaling where the required capacity is known, load-based scaling reacts to metrics, and partial fulfillment might not be desirable.
          Cooldown Time	The interval between the start of an auto scaling activity and the time when another can begin. During this period, all scaling triggers are ignored. After the cooldown ends, the system will only respond to new triggers that meet the scaling conditions. This cooldown period allows system load to stabilize after a scaling event before another one is initiated.
          Effective Time Period	An optional parameter that specifies the time range during which a load-based scaling rule is active. By default, the rule is active 24/7. If you set a time range, scaling activities are triggered only within that range.

   3. After you complete the configuration, click Save and Apply.

      When the conditions are met, auto scaling activities are triggered for the node group.

Method 2: Configure rules during cluster creation

1. Log on to the EMR on ECS console.

2. In the top navigation bar, select a region and resource group based on your requirements.

3. Click CREATE_CLUSTER. For more information about the parameters, see Create a cluster.

   Note

   You must add a pay-as-you-go task node group to the cluster before you can configure the corresponding rules.

4. Configure Cluster Scaling.

   1. Select Custom Auto Scaling Rule and click Edit in the Actions column of the target node group.

   2. In the Configure Auto Scaling panel, configure the parameters. For more information, see Configure scaling rules.

   3. After you complete the configuration, click Save and Apply.

5. Confirm the order and create the cluster.

   After the cluster is created, scaling activities are triggered for the node group when the rule conditions are met.

Method 3: Configure rules with an SDK

When you create a cluster or node group, you can configure custom scaling rules for the node group to automate node creation and management. For more information, see Create a cluster and Create a node group. You can also configure custom scaling rules for an existing node group. For more information, see Create custom auto scaling policies.

The following Java code provides an example of configuring a load-based scale-out rule.

// This file is auto-generated, don't edit it. Thanks.
package com.aliyun.sample;

import com.aliyun.tea.*;

public class Sample {

    /**
     * <b>description</b> :
     * <p>Use an AccessKey pair to initialize the client.</p>
     * @return Client
     * 
     * @throws Exception
     */
    public static com.aliyun.emr20210320.Client createClient() throws Exception {
        // Leaking source code may cause the AccessKey pair to be leaked, posing a threat to the security of all your resources. The following code is for reference only.
        // We recommend that you use the more secure STS method for authentication.
        com.aliyun.teaopenapi.models.Config config = new com.aliyun.teaopenapi.models.Config()
                // Required. Make sure that the ALIBABA_CLOUD_ACCESS_KEY_ID environment variable is set.
                .setAccessKeyId(System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID"))
                // Required. Make sure that the ALIBABA_CLOUD_ACCESS_KEY_SECRET environment variable is set.
                .setAccessKeySecret(System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET"));
        // For the endpoint, see https://api.alibabacloud.com/product/Emr.
        config.endpoint = "emr.eu-central-1.aliyuncs.com";
        return new com.aliyun.emr20210320.Client(config);
    }

    public static void main(String[] args_) throws Exception {
        java.util.List<String> args = java.util.Arrays.asList(args_);
        com.aliyun.emr20210320.Client client = Sample.createClient();
        com.aliyun.emr20210320.models.ScalingConstraints scalingConstraints = new com.aliyun.emr20210320.models.ScalingConstraints()
                .setMaxCapacity(10)
                .setMinCapacity(0);
        com.aliyun.emr20210320.models.TriggerCondition scalingRule0MetricsTriggerTriggerCondition0 = new com.aliyun.emr20210320.models.TriggerCondition()
                .setMetricName("yarn_resourcemanager_queue_AvailableVCoresPercentage")
                .setStatistics("AVG")
                .setComparisonOperator("LE")
                .setThreshold(15D);
        com.aliyun.emr20210320.models.TriggerCondition scalingRule0MetricsTriggerTriggerCondition1 = new com.aliyun.emr20210320.models.TriggerCondition()
                .setMetricName("yarn_resourcemanager_queue_AvailableMBPercentage")
                .setStatistics("AVG")
                .setComparisonOperator("LE")
                .setThreshold(15D);
        com.aliyun.emr20210320.models.MetricsTrigger scalingRule0MetricsTrigger = new com.aliyun.emr20210320.models.MetricsTrigger()
                .setTimeWindow(300)
                .setEvaluationCount(1)
                .setCoolDownInterval(300)
                .setConditionLogicOperator("Or")
                .setConditions(java.util.Arrays.asList(
                    scalingRule0MetricsTriggerTriggerCondition0,
                    scalingRule0MetricsTriggerTriggerCondition1
                ));
        com.aliyun.emr20210320.models.ScalingRule scalingRule0 = new com.aliyun.emr20210320.models.ScalingRule()
                .setRuleName("default")
                .setTriggerType("METRICS_TRIGGER")
                .setActivityType("SCALE_OUT")
                .setAdjustmentValue(1)
                .setMetricsTrigger(scalingRule0MetricsTrigger);
        com.aliyun.emr20210320.models.PutAutoScalingPolicyRequest putAutoScalingPolicyRequest = new com.aliyun.emr20210320.models.PutAutoScalingPolicyRequest()
                .setRegionId("cn-hangzhou")
                .setClusterId("c-xxxx")
                .setNodeGroupId("ng-xxx")
                .setScalingRules(java.util.Arrays.asList(
                    scalingRule0
                ))
                .setConstraints(scalingConstraints);
        com.aliyun.teautil.models.RuntimeOptions runtime = new com.aliyun.teautil.models.RuntimeOptions();
        try {
            // Print the API return values on your own.
            client.putAutoScalingPolicyWithOptions(putAutoScalingPolicyRequest, runtime);
        } catch (TeaException error) {
            // The following code is for demonstration purposes only. We recommend that you handle exceptions with caution and do not ignore them in your project.
            // Error message
            System.out.println(error.getMessage());
            // Diagnosis address
            System.out.println(error.getData().get("Recommend"));
            com.aliyun.teautil.Common.assertAsString(error.message);
        } catch (Exception exception) {
            TeaException error = new TeaException(exception.getMessage(), exception);
            // The following code is for demonstration purposes only. We recommend that you handle exceptions with caution and do not ignore them in your project.
            // Error message
            System.out.println(error.getMessage());
            // Diagnosis address
            System.out.println(error.getData().get("Recommend"));
            com.aliyun.teautil.Common.assertAsString(error.message);
        }        
    }
}