E-MapReduce:Manage YARN partitions in the EMR console

Use cases

Use YARN partition management when:

• Heterogeneous instances exist in your cluster. For example, your cluster mixes compute-optimized instances, memory-optimized instances, and storage-optimized instances. Partitions let you assign workloads to the right instance type.

• Multiple computing frameworks share the cluster. Batch jobs (for example, a Flink offline processing job) can consume large amounts of CPU or network resources. Partitions prevent these jobs from starving near-real-time workloads.

• Critical jobs must be protected from scale-in. Assign inelastic nodes to a dedicated partition so important jobs run there and are never interrupted by a scale-in activity.

Prerequisites

Before you begin, ensure that you have:

• A cluster in the Running state. See Create a cluster

Limitations

• The cluster must run EMR V5.11.1 or later, or EMR V3.45.1 or later.

• The cluster must be in the Running state.

• The capacity scheduler must be used as the YARN scheduler.

• After adding or editing a partition, click Deploy to apply the changes. Perform this operation during off-peak hours.

• If you use the fair scheduler, turn off Node Labels before managing partitions.

Add a partition

1. Go to the Configure tab of the YARN service page.

   1. Log on to the EMR console. In the left-side navigation pane, click EMR on ECS.

   2. In the top navigation bar, select the region where your cluster resides and select a resource group.

   3. Find the cluster and click Services in the Actions column.

   4. On the Services tab, click Configure in the YARN section.

2. Click the Partitions tab.

3. Click Add Partition. In the Add Partition dialog box, configure the following parameters.

   Parameter	Description
   Partition Name	The partition name, 1–255 characters. Allowed characters: letters, digits, hyphens (-), and underscores (_). The name cannot start with - or _. To create the default partition, name it DEFAULT.
   Partition Type	Exclusive (default): containers are allocated to nodes in this partition only when the partition is explicitly requested. Non Exclusive: containers are allocated to nodes in this partition when the partition is explicitly requested. If no partition or the DEFAULT partition is specified, idle resources in this partition are also available to those jobs.
   Associated Node Groups	The node groups to associate with the partition. Each node group can be associated with only one partition. You can modify the associated node groups.

4. Click OK.

5. Click Deploy. After the configurations take effect, the new partition appears on the Partitions tab.

   By default, configurations take up to 10 minutes to propagate. To shorten this window, add the yarn.nodemanager.node-labels.resync-interval-ms and yarn.nodemanager.node-labels.provider.fetch-interval-ms parameters on the yarn-site.xml tab, then manually restart the NodeManager component on the Status tab.

Edit a partition

1. On the Partitions tab of the YARN service page, find the partition and click Edit in the Actions column.

2. In the Edit Partition dialog box, modify the associated node groups. Other parameters cannot be changed.

3. Click OK.

4. Click Deploy.

Delete a partition

1. On the Partitions tab of the YARN service page, find the partition and click Delete in the Actions column.

2. In the confirmation dialog box, click OK.

3. Click Deploy.

   Deleting a partition disassociates all node groups linked to it. If the partition is associated with a queue, turn off Disable Partition-queue Association Management on the Edit Resource Queue tab before deleting.

View partition information

On the Partitions tab of the YARN service page, you can view each partition's name, accessible queues, associated node group, partition type, and total resources.

Disable the partition management feature

To switch the YARN scheduler from the capacity scheduler to the fair scheduler, turn off Node Labels. Then apply the change: click at the top of the Edit Resource Queue tab, select all configuration items in the Configurations to Take Effect dialog box, and click batch restart.

Remove the Partitions tab from the EMR console

To manage partitions with a custom script and disable the EMR console's partition management, set node_labels_managed_by_emr to false on the yarn-operator-conf tab of the Configure tab on the YARN service page. After refreshing the console, the Partitions tab is hidden and partition validation against node-labels.xml is disabled. For details on using the node label feature directly, see Node labels.