E-MapReduce:Manage node groups

Limits

• The operations described in this topic apply only to DataLake, DataFlow, OLAP, DataServing, and Custom clusters.

• You cannot perform Upgrade Configuration on Task node groups for which the Billing Method is Pay-as-you-go or Preemptible Instance.

  For more information about how to upgrade node configurations, see Upgrade node configurations.

Add a node group

1. Go to the node management page.

   1. Log on to the EMR console.

   2. In the top navigation bar, select a region and a resource group as needed.

   3. In the Actions column of the target cluster, click Nodes.

2. On the Nodes page, click Add Node Group.

3. In the Add Node Group pane, configure the following parameters.

   Parameter	Description
   Zone	The zone where the cluster is located is displayed by default. Click View Zones to select another zone in the same region. You can add only Task node groups in other zones. After adding a cross-zone node group, we recommend using the YARN Node Labels feature to partition the cluster. This approach minimizes the impact of unstable network bandwidth on task efficiency, especially during the shuffle process. For more information, see Use Node Labels for node partitioning.
   Node Group Type	You can add the following types of node groups: Core: A core node group. Suitable for use cases with small data volumes, such as log analysis and website traffic statistics. Task: A compute node group. Suitable for use cases that require temporary computing resources, such as batch processing and data cleansing. Gateway: A task node group. This option is available only for DataLake and DataFlow clusters that run EMR-5.10.1 or later and Custom clusters that run EMR-5.17.1 or later. Suitable for use cases that require frequent job submissions, such as model training by data scientists and data processing by data engineers. Master-Extend: A load extension group. This option is available only for high availability clusters that run EMR-3.51.1 or later, or EMR-5.17.1 or later. If the master node of a cluster is under high load, you can add a Master-Extend node group and deploy services to different node groups to reduce the load on the master node. This is suitable for use cases that involve large-scale clusters with high master node loads. Note By default, services are not deployed to a Master-Extend node group. You must select the services to deploy to this group during its creation.
   Billing Method	The billing method for the node group. Supported methods are pay-as-you-go, spot instance, and subscription. Note Only Task node groups support spot instances.
   Node Group Name	Node group names must be unique within the cluster.
   Components	Only Master-Extend node groups support custom service deployment. The following services can be deployed: Hive: HiveMetaStore, HiveServer Kyuubi: KyuubiServer Spark: SparkHistoryServer, SparkThriftServer
   Assign Public Network IP	Specifies whether to assign public IP addresses to the nodes in the node group. If you enable this option, all nodes in the group are assigned a public IP address.
   vSwitch	You can specify a vSwitch within the same VPC when you create a node group. This setting cannot be changed after the node group is created. Note The vSwitch must be in the same zone as the cluster.
   Additional Security Group	(Optional) You can assign additional security groups to the node group. A node group can have up to four additional security groups.
   Instance Type	Select an instance type based on your requirements. If the billing method is subscription, you can select only one instance type. If the billing method is pay-as-you-go or spot instance and the node group type is Task, you can select up to 10 alternative instance types that have the same vCPU and memory specifications.
   Storage Configuration	System Disk: Select an ESSD or an Ultra Disk based on your needs. The system disk size can be 60 to 500 GiB. We recommend a size of at least 120 GiB. Data Disk: Select an ESSD or an Ultra Disk based on your needs. The data disk size can be 40 to 32,768 GiB. We recommend a size of at least 80 GiB. Note If you select enhanced SSDs, you can specify different performance levels (PLs) for the enhanced SSDs based on the disk capacity to meet different cluster performance requirements. The default performance level is PL1. When you configure the system disk, you can select an enhanced SSD of the following performance levels: PL0, PL1, and PL2. When you configure data disks, you can select enhanced SSDs of the following performance levels: PL0, PL1, PL2, and PL3. For more information, see Disks.
   Resource Reservation Policy	Note This parameter is available only when Node Group Type is set to Task and Billing Method is set to Pay-as-you-go. The resource reservation policy allows you to associate your private ECS capacity pools. You can go to the ECS console to reserve resources. For more information, see Resource Butler overview. Public Pool Only (default): Fulfills resource requests directly from public resource pools. Private Pool First: Prioritizes provisioning ECS instances from your private pool. If your private pool has insufficient resources, the system falls back to the public pool. Specified Private Pool: Specifies a private ECS pool for the current E-MapReduce cluster.
   Automatic Compensation	Note This parameter is available only when Node Group Type is set to Task. If this feature is enabled, E-MapReduce monitors the nodes in the node group for health issues. If any nodes become unhealthy, E-MapReduce releases them and creates an equal number of replacement nodes. For more information, see Node compensation.
   Scaling Policy	Note This parameter can be configured only when Billing Method is set to Spot Instance. Priority-based Policy (default) The system attempts to create an instance by trying each specified instance type in order until one is successfully created. The final instance type purchased may vary depending on inventory. Cost Optimization Policy During a scale-out, auto scaling attempts to create ECS instances in ascending order of vCPU unit price. During a scale-in, it removes ECS instances in descending order of vCPU unit price. If the billing method for the scaling configuration is set to spot instance, the system prioritizes creating spot instances. If spot instances of the specified types cannot be created due to inventory or other reasons, the system automatically attempts to create pay-as-you-go instances instead. For more information, see Cost optimization mode.
   Graceful Shutdown	Note This parameter is available only for clusters where the YARN service is deployed. If enabled, during a node group scale-in, the system waits for tasks on a node to complete or for a specified timeout period to elapse before removing the node. You can modify the graceful shutdown timeout by configuring the yarn.resourcemanager.nodemanager-graceful-decommission-timeout-secs parameter on the YARN service page.
   Add to Deployment Set	Note This parameter is available for Core, Gateway, and Master-Extend node groups. This feature is disabled by default. If you enable this feature, ECS instances are deployed on different physical servers to ensure high availability. For more information, see Enable a deployment set.
   Custom Software Configuration	This feature is disabled by default. If you enable this feature, you can provide a JSON configuration file to add node group-level parameters for services during node group creation. For more information, see Configure custom software.

   If you do not specify the Minimum Pay-As-You-Go Nodes, Percentage of Pay-As-You-Go Nodes, or Lowest-Cost Instance Types parameter, the machine group is a general cost optimization scaling group. If you specify the parameters, the machine group is a mixed-instance cost optimization scaling group. The two types of cost optimization scaling groups are fully compatible with each other in terms of interfaces and features.

   You can use a mixed-instance cost optimization scaling group to achieve the same effect as a specific general cost optimization scaling group by configuring appropriate mixed-instance policies. Examples:

   • In a general cost optimization scaling group, only pay-as-you-go instances are created.

     In your mixed-instance cost optimization scaling group, set Minimum Pay-As-You-Go Nodes to 0, Percentage of Pay-As-You-Go Nodes to 100, and Lowest-Cost Instance Types to 1.

   • In a general cost optimization scaling group, preemptible instances are preferentially created.

     In your mixed-instance cost optimization scaling group, set Minimum Pay-As-You-Go Nodes to 0, Percentage of Pay-As-You-Go Nodes to 0, and Lowest-Cost Instance Types to 1.

4. Click OK.

   After the node group is added, you can view it on the Nodes page.

Modify a node group

1. On the Nodes page, click the name of the target node group in the Node Group Name column.

2. In the Node Group Attributes dialog box, modify the node group settings and click Save.

   • For Master, Core, Gateway, and Master-Extend node groups, you can modify the node group name and additional security groups.

   • For Task node groups, you can modify information such as the node group name, node specifications, and additional security groups. You can also adjust the settings in the Advanced Information section.

Delete a node group

1. On the Nodes page, find the target node group and click Delete Node Group in the Actions column.

2. In the confirmation dialog box that appears, click Delete.

Cost optimization mode

This mode allows you to define more detailed cost control policies to balance cost and stability.

Related topics

• To scale out a node group, see Scale out a cluster.

• To scale in a node group, see Scale in a cluster.

• To expand disk capacity, see Expand a disk.

• To configure auto scaling policies, see Create a custom auto scaling policy.

• To view auto scaling activities, see View auto scaling activities.