Resource queues are used to manage and schedule cluster resources for flexible allocation and optimization. In the EMR console, you can manage resource queues and associate them with different partitions.
Prerequisites
A cluster has been created and is running. For more information about how to create a cluster, see Create a cluster.
Limitations
-
The YARN scheduler must be the Capacity Scheduler.
-
Custom partition features are available for EMR-5.11.1 or later and EMR-3.45.1 or later.
Associate YARN partitions with queues
-
This feature is available for EMR-5.12.1 or later and EMR-3.46.1 or later.
-
If you disable the partition-queue association, the system automatically disassociates all queues from the current partition.
-
Go to the configuration page of the YARN service.
-
Log on to the E-MapReduce console.
-
In the top navigation bar, select a region and resource group.
-
On the EMR on ECS page, find your target cluster and click Services in the Actions column.
-
On the Services page, click Configure in the YARN section.
-
-
Click the Edit Resource Queue tab.
-
From the Select Partition drop-down list, select an existing partition.
By default, the
DEFAULT_PARTITIONpartition is selected, and all queues can access this partition. To learn how to add a partition, see Manage YARN partitions in the EMR console. -
Turn on the Enable Partition-queue Association Management switch.
A list of queues that can access the current partition or have capacity in the partition appears.
-
In the dialog box that appears, set the Queue Capacity and click OK.
-
Apply the configurations.
-
Click the
icon above the Edit Resource Queue tab. -
In the dialog box that appears, find the relevant configuration file and click refresh_queues in the Actions column.
-
Add a queue
To add a child queue to a leaf queue, you must first stop the leaf queue. Otherwise, the deployment fails.
-
(Optional) On the Edit Resource Queue tab of the YARN service, select an existing partition from the Select Partition drop-down list.
By default, the
DEFAULT_PARTITIONpartition is displayed. -
Click Manage Queues or Manage Child Queues.
-
Select a parent queue from the Select Parent Queue drop-down list.
The root queue is selected by default.
-
In the Manage Child Queues dialog box, click Add Queue.
-
Set Queue Name, Queue Capacity, and Status, and then click OK.
The combined capacity of all child queues under the same parent queue must equal 100%.
-
Apply the configurations.
-
Click the
icon above the Edit Resource Queue tab. -
In the dialog box that appears, find the relevant configuration file and click refresh_queues in the Actions column.
-
Edit a queue
-
(Optional) On the Edit Resource Queue tab of the YARN service, select an existing partition from the Select Partition drop-down list.
By default, the
DEFAULT_PARTITIONpartition is displayed. -
On the Edit Resource Queue tab, click the
icon next to the root queue. -
In the Actions column of the target queue, click Edit.
-
In the Edit dialog box, modify the following parameters and click OK.
Parameter
Description
Current Queue Name
The name of the queue. You can customize the name. To edit the name, go to Manage Queues or Manage Child Queues.
Minimum User Capacity
The percentage of cluster resources that a single user can use. This parameter corresponds to
Configured Minimum User Limit Percenton the YARN web UI.This parameter can be edited only for the
DEFAULT_PARTITIONpartition.User Limit Factor
The ratio of the maximum resources that a single user can use to the queue's capacity. This parameter corresponds to
Configured user limit factoron the YARN web UI.This parameter can be edited only for the
DEFAULT_PARTITIONpartition.Resource Queue Status
The status of the resource queue. Valid values: Started and STOP. This parameter can be edited only for the
DEFAULT_PARTITIONpartition.NoteBefore you set a parent queue to Stopped, you must stop all its child queues.
Accessible Partitions
The partitions that the queue can access.
Note-
For EMR-5.12.1 or later and EMR-3.46.1 or later, all queues can access the
DEFAULT_PARTITIONpartition by default. -
A child queue inherits the accessible partitions of its parent queue by default, but you can configure them separately for the child queue.
Default Partition
The default partition for the queue. If you do not configure a specific partition, the queue defaults to using
DEFAULT_PARTITION. -
-
Apply the configurations.
-
Click the
icon above the Edit Resource Queue tab. -
In the dialog box that appears, find the relevant configuration file and click refresh_queues in the Actions column.
-
Delete a queue
-
The queue must be in the STOP state to be deleted. Otherwise, the deployment will fail.
-
Before you delete a queue, make sure that its capacity is 0 in all partitions.
-
The combined capacity of all child queues under the same parent queue must equal 100%.
-
If you use YARN version 2.8.5, the refresh_queues operation will not take effect when you delete a queue. You must manually restart ResourceManager on the YARN Status tab for the changes to take effect.
-
(Optional) On the Edit Resource Queue tab of the YARN service, select an existing partition from the Select Partition drop-down list.
By default, the
DEFAULT_PARTITIONpartition is displayed. -
Click Manage Queues.
-
In the Manage Child Queues dialog box, set the STOP of the target queue to Stopped and its Queue Capacity to 0. Then, click OK.
-
Apply the configurations.
-
Click the
icon above the Edit Resource Queue tab. -
In the dialog box that appears, find the relevant configuration file and click refresh_queues in the Actions column.
-
-
Click Manage Queues again.
-
In the Manage Child Queues dialog box, find the queue that you want to delete, click Delete in the Actions column, and then click OK.
-
Apply the configurations.
-
Click the
icon above the Edit Resource Queue tab. -
In the dialog box that appears, find the relevant configuration file and click refresh_queues in the Actions column.
-