E-MapReduce:YARN schedulers

Scheduling modes

Capacity Scheduler's MainScheduler supports three triggering modes:

Global scheduling requires all asynchronous scheduling configurations in addition to its own settings. See Global scheduling for details.

Architecture (global scheduling)

The following figure shows the architecture of global scheduling based on YARN 3.2 or later.

MainScheduler runs as an asynchronous multi-threaded framework:

• Allocation threads identify the highest-priority resource request, select candidate nodes based on resource size and placement constraints, generate allocation proposals, and place them in an intermediate queue.

• Submission thread consumes allocation proposals, rechecks placement constraints and application/node requirements, then either commits or discards each proposal and updates scheduler state.

ReScheduler runs periodically as a dynamic resource monitoring framework. It implements policies for inter-queue preemption, intra-queue preemption, and reserved resource preemption.

Node Sorting Manager and Placement Constraint Manager are global scheduling plug-ins for MainScheduler. They handle load balancing and complex placement constraint management.

Container allocation process

The following figure shows the global scheduling process of Capacity Scheduler.

MainScheduler allocates containers in six steps:

1. Select partitions (node labels). A cluster may have one or more partitions. MainScheduler allocates containers to partitions in turn.

2. Select leaf queues. Traversing from the root queue downward, child queues at each level are visited in ascending order of their guaranteed resource percentage. Queues with lower utilization (shown in green in the figure above) receive resources before those with higher utilization (shown in red).

3. Select applications. Within a queue, MainScheduler selects applications based on the queue's ordering policy:

   • Fair policy: ascending order of allocated memory resources

   • FIFO policy: descending order of priority, then ascending order of application ID

4. Select requests. MainScheduler selects requests within the chosen application based on priority.

5. Select sorted candidate nodes. MainScheduler searches all sorted nodes for candidates that satisfy the request's resource requirements and placement constraints.

6. Allocate a container. For each candidate node, MainScheduler checks the queue's and node's allocated, in-use, and unconfirmed resources. If the check passes, it generates an allocation proposal and places it in the proposal queue.

After allocation, the submission thread verifies each proposal against application, node, and placement constraint requirements. Proposals that fail verification are discarded; approved proposals take effect and update resource accounting for the application and node.

Preemption

ReScheduler monitors cluster resources and triggers preemption when total available resources fall below a threshold and specific applications are under-resourced.

Global configurations

Configure the following settings in yarn-site.xml and capacity-scheduler.xml.

Node heartbeat configurations

Asynchronous scheduling

Global scheduling

Global scheduling requires all asynchronous scheduling configurations listed in the previous table.

Node partition configurations

For node partition (node label) configurations, see Node labels.

Queue configurations

Basic queue configurations

Basic queue configurations define the queue hierarchy, guaranteed capacity, and maximum capacity. The following example shows a root queue with four child queues shared across an organization's departments.

Example queue structure:

Queue	Guaranteed resources	Maximum resources	Child queues
root	100%	100%	dev, test, support, default
root.dev	50%	100%	training, services
root.dev.training	40% of dev	100% of dev	—
root.dev.services	60% of dev	100% of dev	—
root.test	30%	50%	—
root.support	10%	30%	—
root.default	10%	100%	—

The following capacity-scheduler.xml example configures this structure. Copy and adjust it to match your organization's queue layout.

<configuration>
  <!-- Root-level child queues -->
  <property>
    <name>yarn.scheduler.capacity.root.queues</name>
    <value>dev,test,support,default</value>
  </property>

  <!-- dev queue: 50% guaranteed, 100% max -->
  <property>
    <name>yarn.scheduler.capacity.root.dev.capacity</name>
    <value>50</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.dev.maximum-capacity</name>
    <value>100</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.dev.queues</name>
    <value>training,services</value>
  </property>

  <!-- dev.training: 40% of dev guaranteed, 100% of dev max -->
  <property>
    <name>yarn.scheduler.capacity.root.dev.training.capacity</name>
    <value>40</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.dev.training.maximum-capacity</name>
    <value>100</value>
  </property>

  <!-- dev.services: 60% of dev guaranteed, 100% of dev max -->
  <property>
    <name>yarn.scheduler.capacity.root.dev.services.capacity</name>
    <value>60</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.dev.services.maximum-capacity</name>
    <value>100</value>
  </property>

  <!-- test queue: 30% guaranteed, 50% max -->
  <property>
    <name>yarn.scheduler.capacity.root.test.capacity</name>
    <value>30</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.test.maximum-capacity</name>
    <value>50</value>
  </property>

  <!-- support queue: 10% guaranteed, 30% max -->
  <property>
    <name>yarn.scheduler.capacity.root.support.capacity</name>
    <value>10</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.support.maximum-capacity</name>
    <value>30</value>
  </property>

  <!-- default queue: 10% guaranteed, 100% max -->
  <property>
    <name>yarn.scheduler.capacity.root.default.capacity</name>
    <value>10</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.default.maximum-capacity</name>
    <value>100</value>
  </property>
</configuration>

The YARN Scheduler page shows the queue hierarchy, guaranteed resource percentages, and maximum resource percentages. The solid gray box represents the maximum resource percentage; the dashed box represents the guaranteed resource percentage. Expand a queue to view its status, resource usage, application details, and other settings.

The following table describes the capacity-scheduler.xml parameters for this structure.

Configuration file	Configuration item	Sample value	Description
capacity-scheduler.xml	yarn.scheduler.capacity.root.queues	dev,test,support,default	Child queues of the root queue. Separate multiple queues with commas.
	yarn.scheduler.capacity.root.dev.capacity	50	Guaranteed resource percentage for the dev queue relative to total cluster resources.
	yarn.scheduler.capacity.root.dev.maximum-capacity	100	Maximum resource percentage for the dev queue relative to total cluster resources.
	yarn.scheduler.capacity.root.dev.queues	training,services	Child queues of the dev queue.
	yarn.scheduler.capacity.root.dev.training.capacity	40	Guaranteed resource percentage for training relative to dev's guaranteed resources.
	yarn.scheduler.capacity.root.dev.training.maximum-capacity	100	Maximum resource percentage for training relative to dev's maximum resources.
	yarn.scheduler.capacity.root.dev.services.capacity	60	Guaranteed resource percentage for services relative to dev's guaranteed resources.
	yarn.scheduler.capacity.root.dev.services.maximum-capacity	100	Maximum resource percentage for services relative to dev's maximum resources.
	yarn.scheduler.capacity.root.test.capacity	30	Guaranteed resource percentage for test relative to total cluster resources.
	yarn.scheduler.capacity.root.test.maximum-capacity	50	Maximum resource percentage for test relative to total cluster resources.
	yarn.scheduler.capacity.root.support.capacity	10	Guaranteed resource percentage for support relative to total cluster resources.
	yarn.scheduler.capacity.root.support.maximum-capacity	30	Maximum resource percentage for support relative to total cluster resources.
	yarn.scheduler.capacity.root.default.capacity	10	Guaranteed resource percentage for default relative to total cluster resources.
	yarn.scheduler.capacity.root.default.maximum-capacity	100	Maximum resource percentage for default relative to total cluster resources.

Preemption configurations

Preemption ensures fair resource distribution across tenants and respects application priorities. Enable it for clusters with strict scheduling requirements. Preemption requires YARN V2.8.0 or later.

Manage queue configurations with the RESTful API

Starting from YARN V3.2.0, you can use the RESTful API to apply incremental configuration updates and view all currently active settings in capacity-scheduler.xml. Earlier versions only support full updates via the RefreshQueues operation, with no way to inspect active configurations.

To enable RESTful API queue management, add the following settings to yarn-site.xml:

View active configurations:

• RESTful API: http://<rm-address>/ws/v1/cluster/scheduler-conf

• HDFS: ${yarn.scheduler.configuration.fs.path}/capacity-scheduler.xml.<timestamp> — the file with the largest timestamp is the latest version.

Update configurations incrementally:

The following example changes yarn.scheduler.capacity.maximum-am-resource-percent to 0.2 and deletes the yarn.scheduler.capacity.xxx parameter. To delete a parameter, omit the value field.

curl -X PUT -H "Content-type: application/json" 'http://<rm-address>/ws/v1/cluster/scheduler-conf' -d '
{
  "global-updates": [
    {
      "entry": [{
        "key": "yarn.scheduler.capacity.maximum-am-resource-percent",
        "value": "0.2"
      },{
        "key": "yarn.scheduler.capacity.xxx"
      }]
    }
  ]
}'

Resource limits per container

The maximum resources for a single container are determined by the following settings. If a container request exceeds the limit, the scheduler logs an InvalidResourceRequestException: Invalid resource request... error.