E-MapReduce:Configure custom auto scaling rules

Prerequisites

Before you begin, make sure that you have:

• A DataLake, Dataflow, online analytical processing (OLAP), DataServing, or custom cluster

• A task node group with pay-as-you-go or preemptible instances in the cluster. For details, see Create a node group

Limitations

• You can add up to 10 Elastic Compute Service (ECS) instance types per node group. When the cluster scales out, EMR tries each instance type in order, starting with the first one. If an instance type is unavailable, EMR falls back to the next type.

• Only clusters with the YARN service deployed support load-based scaling rules.

Usage notes

• Auto scaling rules are applied per node group. If no rule is configured for a node group, that node group never scales automatically.

• EMR automatically finds and lists ECS instance types that match your specified instance specifications. Select at least one instance type from the list so that the cluster can scale based on that type.

Choose a scaling type

Before configuring rules, identify which scaling type fits your workload:

Only clusters with YARN deployed support load-based scaling. If your cluster does not have YARN, use time-based scaling.

Configure auto scaling rules

// This file is auto-generated, don't edit it. Thanks.
package com.aliyun.sample;

import com.aliyun.tea.*;

public class Sample {

    /**
     * description

Auto scaling parameters

Descriptions of EMR auto scaling metrics that match YARN load metrics

For queue-related metrics, queue_name specifies the queue name. The default value is root. Specify a custom queue name as needed.

For partition-related metrics, partition_name specifies the partition name. This field cannot be left blank.

Troubleshooting

Scale-out fails or succeeds only partially

Scale-out failures typically have two causes:

• ECS resource shortage: The requested instance type has insufficient inventory in the selected zone. ECS resource availability is dynamic and can be temporarily exhausted during high-demand periods.

• Single instance type configured: If only one instance type is set for the node group and that type runs out, the entire scale-out request fails.

To improve scale-out reliability:

• Enable Best-effort Delivery: When only some of the requested instances are available, EMR delivers those instead of failing the whole operation.

• Configure multiple instance types: Add up to 10 instance types to the task node group. EMR tries each type in order — if the first is unavailable, it falls back to the next. For details on adding instance types, see Create a node group.

What's next

• To monitor scaling activity, check the Auto Scaling tab of your cluster.

• To create or modify the task node group where rules are applied, see Create a node group.

• To programmatically manage auto scaling policies, see PutAutoScalingPolicy.