AnalyticDB for MySQL Data Lakehouse Edition (V3.0) provides the priority queue feature for jobs in job resource groups. Each job resource group has a set of priority queues, which are LOWEST, LOW, NORMAL, and HIGH. You can configure query priorities to allow queries to enter different priority queues. This topic describes how to configure and view the job priorities of job resource groups.
Prerequisites
An AnalyticDB for MySQL Data Lakehouse Edition (V3.0) cluster of V3.1.6.3 or later is created.
A job resource group is created. A job is submitted to run in the job resource group.
Usage notes
You cannot modify the priorities of submitted jobs regardless of whether the jobs are running.
Overview
Job priorities determine which jobs in a queue are preferentially allocated with resources. If a large number of jobs are concurrently run in a job resource group of an AnalyticDB for MySQL Data Lakehouse Edition (V3.0) cluster, the jobs require more resources than the capacity of the resource group. AnalyticDB for MySQL Data Lakehouse Edition (V3.0) supports the job queue feature. If resources are insufficient, new jobs are queued and remain in the submitted state.
For example, if a key metric depends on the intermediate result of an upstream job, you must configure a higher priority for the upstream job to provide sufficient resources for the job. AnalyticDB for MySQL Data Lakehouse Edition (V3.0) provides the priority queue feature for jobs in job resource groups. You can configure job priorities to allow jobs to enter the following priority queues: LOWEST, LOW, NORMAL, and HIGH. Jobs that have higher priorities are preferentially run.
Configure job priorities
You can specify the following priorities for jobs: HIGH, NORMAL, LOW, and LOWEST. If you do not specify a priority for a job, the job is assigned the NORMAL priority.
Extract-transform-load (ETL) queries and SELECT queries are of the NORMAL priority.
The configuration method of job priorities varies based on the job type. The following section describes how to configure a job priority:
Configure a priority for XIHE BSP jobs
Add a hint: /*query_priority= <priority level>*/
Example: Set the priority of XIHE bulk synchronous parallel (BSP) jobs to HIGH to allow the jobs to enter the HIGH queue.
/*query_priority=HIGH*/ SELECT * FROM test_table;Configure a priority for Spark SQL jobs
Execute the SET statement: SET spark.adb.priority = <priority level>
Example: Set the priority of Spark SQL jobs to LOW to allow the jobs to enter the LOW queue.
SET spark.adb.priority = LOW;
SELECT * FROM test_table;Configure a priority for Spark batch jobs
Configure the conf parameter: "conf": { "spark.adb.priority": "<priority level>"}
Example: Set the priority of Spark batch jobs to HIGH to allow the jobs to enter the HIGH queue.
{
"comments": [
"-- Here is just an example of SparkPi. Modify the content and run your spark program."
],
"args": [
"1000"
],
"file": "local:///tmp/spark-examples.jar",
"name": "SparkPi",
"className": "org.apache.spark.examples.SparkPi",
"conf": {
"spark.driver.resourceSpec": "medium",
"spark.executor.instances": 2,
"spark.executor.resourceSpec": "medium",
"spark.adb.priority": "HIGH"
}
}View job priorities
View the priorities of XIHE BSP and Spark SQL jobs
Log on to the AnalyticDB for MySQL console. In the left-side navigation pane of a cluster, choose Job Development > SQL Development. On the Execution Records tab, view the priorities of XIHE BSP and Spark SQL jobs.
View the priorities of Spark batch jobs
Log on to the AnalyticDB for MySQL console. In the left-side navigation pane of a cluster, choose Job Development > Spark JAR Development. On the Applications tab, view the priorities of Spark batch jobs.
