E-MapReduce:Manage YARN queues

Prerequisites

Before you begin, ensure that you have:

• A cluster in the Running state. For details, see Create a cluster

• Capacity Scheduler configured as the YARN scheduler

Limitations

• Custom partition features require EMR V5.11.1, EMR V3.45.1, or a later minor version.

Associate a resource queue with a YARN partition

A YARN partition restricts where jobs in a queue can run—either to a specific set of labeled nodes (exclusive) or including those nodes when idle (non-exclusive). Associating a resource queue with a partition controls which queues can consume resources from that partition.

1. Go to the Edit Resource Queue tab.

   1. Log on to the EMR console. In the left-side navigation pane, click EMR on ECS.

   2. In the top navigation bar, select the region where your cluster resides and select a resource group.

   3. On the EMR on ECS page, find the target cluster and click Services in the Actions column.

   4. On the Services tab, click Configure in the YARN section.

   5. Click the Edit Resource Queue tab.

2. From the Select Partition drop-down list, select a partition. The default partition is DEFAULT_PARTITION, which all resource queues can access. To add a partition, see Manage YARN partitions in the EMR console.

3. Turn on Enable Partition-queue Association Management. Queues that can access the current partition or have capacity in it are displayed.

4. In the Enable Partition-queue Association Management dialog box, set Queue Capacity and click OK.

5. Apply the configuration.

   1. Click the icon at the top of the Edit Resource Queue tab.

   2. In the dialog box, find the target component and click refresh_queues in the Actions column.

Add a resource queue

1. On the Edit Resource Queue tab, optionally select a partition from the Select Partition drop-down list. The default partition is DEFAULT_PARTITION.

2. Click Manage Queues, or click Manage Child Queues in the Actions column of the target queue.

3. In the Manage Child Queues dialog box, select a parent queue from the Select Parent Queue drop-down list. The default parent queue is the root queue.

4. Click Add Queue.

5. Set Queue Name, Queue Capacity, and Status, then click OK. The combined capacity of all child queues under a parent queue must equal 100%.

6. Apply the configuration.

   1. Click the icon at the top of the Edit Resource Queue tab.

   2. In the dialog box, find the target component and click refresh_queues in the Actions column.

Modify a resource queue

1. On the Edit Resource Queue tab, optionally select a partition from the Select Partition drop-down list. The default partition is DEFAULT_PARTITION.

2. Click the icon to the left of the root queue name to expand the queue tree.

3. Find the target queue and click Edit in the Actions column.

4. In the Edit dialog box, modify the parameters and click OK.

   Parameter	Description
   Current queue name	The name of the resource queue. Edit it via Manage Queues or Manage Child Queues in the Actions column.
   Minimum user capacity	The maximum percentage of cluster resources a single user can consume in this queue. Corresponds to Configured Minimum User Limit Percent on the YARN web UI. Editable only in DEFAULT_PARTITION.
   User limit factor	The ratio of the maximum resources a single user can use to the total queue capacity. Corresponds to Configured user limit factor on the YARN web UI. Editable only in DEFAULT_PARTITION.
   Resource queue status	The queue status. Valid values: Started and Stopped. Editable only in DEFAULT_PARTITION. To stop a parent queue, stop all its child queues first.
   Accessible partitions	The partitions this queue can access. By default, all queues can access DEFAULT_PARTITION in EMR V5.12.1, EMR V3.46.1, or a later minor version. A child queue inherits its parent's accessible partitions but can also be configured separately.
   Default partition	The default partition for this queue. If not set, DEFAULT_PARTITION is used.

5. Apply the configuration.

   1. Click the icon at the top of the Edit Resource Queue tab.

   2. In the dialog box, find the target component and click refresh_queues in the Actions column.

Delete a resource queue

Deleting a queue is a two-phase process: first prepare the queue for deletion and apply the change, then delete it and apply again.

Before you start: A queue can be deleted only when all of the following conditions are met:

• The queue is in the Stopped state.

• The queue's capacity is 0 in all partitions.

• The combined capacity of remaining child queues in the parent queue equals 100%.