Set priority for jobs in a job resource group - AnalyticDB

Prerequisites

An AnalyticDB for MySQL Enterprise Edition, Basic Edition, or Data Lakehouse Edition cluster is created.
Your cluster's kernel version must be 3.1.6.3 or later.

Note
To view and update the minor version, go to the Configuration Information section on the Cluster Information page in the AnalyticDB for MySQL console.
You must create and select a job resource group when you submit jobs.

Limitations

You cannot change the priority of a submitted job. This applies to both pending and running jobs.

Overview

Job priority determines the order in which queued jobs are allocated resources. When multiple jobs run concurrently in a job resource group, the required resources may exceed the group's available resources. AnalyticDB for MySQL supports job queuing, which queues new jobs instead of letting them fail.

In some business scenarios, the order of job execution and result generation is critical. For example, if the generation of key metrics depends on the output time of an upstream intermediate result, you need to set a high priority for the upstream processing job to ensure that it has sufficient resources. AnalyticDB for MySQL provides priority queues within job resource groups. You can set a job priority to place jobs into different priority queues, which include the LOWEST, LOW, NORMAL, and HIGH queues. Jobs with a higher priority run first.

Set job priority

Valid values for job priority are HIGH, NORMAL, LOW, and LOWEST. If you do not specify a priority, the job priority defaults to NORMAL.

By default, extract, transform, and load (ETL) queries and standard SELECT queries have a NORMAL priority.

The method for setting the job priority varies by job type.

XIHE BSP

Use a hint: /*+ query_priority= <priority_value>*/

Example: Set the job priority for an XIHE BSP job to HIGH to place the job in the HIGH queue.

/*+ query_priority=HIGH*/ SELECT * FROM test_table;

Spark SQL

Use a SET statement: SET spark.adb.priority = <priority_value>

Example: Set the job priority for a Spark SQL job to LOW to place the job in the LOW queue.

 SET spark.adb.priority = LOW;
 SELECT * FROM test_table;

Spark batch

Configure the conf parameter: "conf": { "spark.adb.priority": "<priority_value>"}

Example: Set the job priority for a Spark batch job to HIGH to place the job in the HIGH queue.

{
    "comments": [
        "-- Here is just an example of SparkPi. Modify the content and run your spark program."
    ],
    "args": [
        "1000"
    ],
    "file": "local:///tmp/spark-examples.jar",
    "name": "SparkPi",
    "className": "org.apache.spark.examples.SparkPi",
    "conf": {
        "spark.driver.resourceSpec": "medium",
        "spark.executor.instances": 2,
        "spark.executor.resourceSpec": "medium",
        "spark.adb.priority": "HIGH"
    }
}

View job priority

XIHE BSP and Spark SQL

Log on to the AnalyticDB for MySQL console and navigate to the Execution Records tab on the Job Development > SQL Development page to view job priorities.

Spark batch

Log on to the AnalyticDB for MySQL console and navigate to the Applications tab on the Job Development > Spark JAR Development page to view job priorities.