All Products
Search

This Product

    E-MapReduce:Job priority

    Document Center

    E-MapReduce:Job priority

    Last Updated:Jun 23, 2026

    When multiple jobs in the same queue request resources at the same time, you can set job priorities to ensure that high-priority jobs receive resources first.

    Prerequisites

    • You have created an EMR Serverless Spark workspace. For more information, see Create a workspace.

    Note

    The job priority feature is available only for allowlisted queues. To enable this feature, submit a ticket to the EMR Serverless Spark team.

    Background information

    When multiple jobs in the same queue compete for resources, the system allocates resources based on the following rules:

    • Priority scheduling: Higher-priority jobs receive resources first.

    • Conditions for resource requests: A job can acquire resources only when the queue has enough available resources to meet its driver requirements. If resources are insufficient, the job waits in the queue without blocking other jobs whose requirements can be met.

    • Dynamic priority inheritance: If a job has dynamic resource allocation enabled, any new executors it requests during runtime inherit the original job's priority.

    Priority details

    • Value range: An integer from 0 to 9. A higher value indicates a higher priority.

    • Default value: 0, if no priority is explicitly set.

    • Applicable to:

      • Customizable priority: batch jobs, streaming jobs, and batch or session jobs submitted through a gateway.

      • System-default priority of 5 (customization not supported): session resources, Livy Gateway, and Kyuubi Gateway.

    Procedure

    EMR Serverless Spark supports the following four methods for setting job priority:

    Method 1: Set priority in Data development

    1. Log on to the E-MapReduce console.

      1. In the left-side navigation pane, choose EMR Serverless > Spark.

      2. On the Spark page, click the name of your target workspace.

      3. On the EMR Serverless Spark page, click Data development in the left-side navigation pane.

    2. Create or edit a batch job or streaming job.

    3. In the Spark configuration section of the job, add the following configuration: spark.emr.serverless.priority=9

      In this example, 9 is the priority value. Valid values range from 0 to 9. Set the value based on your business needs.

    4. Save and submit the job.

    Method 2: Set priority through a gateway

    When you submit a job through a Livy Gateway or Kyuubi Gateway, add the priority parameter to the Spark configuration.

    • Livy Gateway: Add the following parameter to the Spark configuration when you submit the job: spark.emr.serverless.priority=8.

    • Kyuubi Gateway: Add the following parameter to the Spark configuration when you submit the job: spark.emr.serverless.priority=8.

    Method 3: Set priority with spark-submit

    When you use the spark-submit command to submit a job, add the priority configuration by using the --conf parameter:

    spark-submit \
  --master spark://<gateway-endpoint> \
  --conf spark.emr.serverless.priority=6 \
  --class org.apache.spark.examples.SparkPi \
  s3://your-bucket/spark-examples.jar 100

    Method 4: Set priority in a workflow

    • Batch job node: Inherits the priority from the Spark configuration of the corresponding job in Data development. No additional configuration is needed.

    • SQL node or Notebook node: In the Spark configuration of the node, add the parameter spark.emr.serverless.priority=7. Then, save and submit the workflow.