Capacity Scheduler is a built-in scheduler in Apache YARN. The YARN service deployed in E-MapReduce (EMR) uses Capacity Scheduler as the default scheduler. Capacity Scheduler is a multi-tenant, hierarchical resource scheduler. Resources used by each child queue are allocated based on the configured capacity.

Enable Capacity Scheduler

Notice After you turn on Enable Resource Queue, you can no longer configure data on the capacity-scheduler tab in the Service Configuration section of the Configure tab on the YARN service page. Existing configurations are synchronized to the Cluster Resources page. If you want to configure cluster resources on the Configure tab of the YARN service page, turn off Enable Resource Queue on the Cluster Resources page.
  1. Log on to the Alibaba Cloud E-MapReduce console.
  2. In the top navigation bar, select the region where your cluster resides.
  3. Click the Cluster Management tab.
  4. On the Cluster Management page that appears, find the target cluster and click Details in the Actions column.
  5. In the left-side navigation pane of the Cluster Overview page that appears, click Cluster Resources.
  6. On the Cluster Resources page that appears, turn on Enable Resource Queue.
  7. Click Save.
    Capacity Scheduler is enabled.

Configure Capacity Scheduler

After Enable Resource Queue is turned on, follow these steps to configure Capacity Scheduler:

  1. On the Cluster Resources page, click Queue Settings in the upper-right corner.
  2. On the Queue Settings tab, find the target queue and click Edit in the Actions column.

    Select Create Child Queue from the More drop-down list in the Actions column to create a child queue.

    root is a level-1 queue. It is the parent queue of all other queues and manages all resources of YARN. By default, only the default queue is available under the root queue.

    Notice
    • The total value of Capacity for all child queues at the same level under the same parent queue must be 100. For example, two child queues default and department are available under the root queue. The total value of Capacity set for these two child queues must be 100. Child queues market and dev are available under the department queue. The total value of Capacity for these two child queues must also be 100.
    • If you do not specify a queue during the running of applications, jobs are submitted to the default queue.
    • If you create level-2 queues under root, you can click Deploy without the need to restart ResourceManager.
    • If you create or set level-3 queues under root, you need to restart ResourceManager.
    • You must restart ResourceManager after you modify the name of a queue.

Switch the scheduler type

To switch the scheduler type after Enable Resource Queue is turned on, follow these steps:

  1. On the Cluster Resources page, click Select Scheduler in the upper-right corner.
  2. Select the required scheduler.
  3. Click Save.
  4. Select RestartResourceManager from the Actions drop-down list in the upper-right corner.
  5. In the Cluster Activities dialog box that appears, specify related parameters and click OK.
  6. In the Confirm dialog box that appears, click OK.
    When a success message is displayed, the scheduler type is switched.

Submit a job

  • If you do not specify a queue during the running of applications, jobs are submitted to the default queue.
  • You must specify a child queue. Tasks cannot be submitted to the parent queue.
  • You must use mapreduce.job.queuename to specify the queue to which you want to submit a job. Example:
    `hadoop jar /usr/lib/hadoop-current/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.5.jar pi -Dmapreduce.job.queuename=test  2 2`