Use the job O&M feature in MaxCompute to view historical and running jobs, review job execution details, analyze the resource load during runtime, and manage your jobs.
Features
The job operations feature in MaxCompute allows you to view and manage both historical and running jobs.
For data developers, this feature simplifies daily tasks such as reviewing job details, identifying exceptions and issues, and responding to problematic jobs by terminating or batch terminating them.
For administrators, this feature helps monitor the resource load of a quota group, enabling them to effectively manage system resources and improve job execution performance.
On the Jobs page in the MaxCompute console, you can configure filter conditions to locate target jobs, view their details, and perform analysis. The specific functions are as follows:
Operations
Filter jobs
Filter jobs by specifying parameters. For a description of the supported parameters, see Job parameter description.
Sort jobs
By default, jobs are sorted by their end time in descending order, with unfinished jobs listed first. You can perform a single-column sort or an advanced multi-column sort.
Single-column sort: You can sort any column that has a sort icon in ascending or descending order.
Advanced multi-column sort: Click the Advanced Sorting button in the upper-right corner of the list. Click Add Sort to add multiple column names, specify the sort order for each column as Ascend or Descend, and then click OK to apply the multi-column sort.
When an advanced sort is active, single-column sorting is disabled. To re-enable single-column sorting, click the Advanced Sorting button, click Reset, and then click OK.
View job details
In the job list, click LogView in the Actions column for a job to open the LogView page, where you can view its execution status, details, and results.
Terminate jobs
You can Terminate or Stop jobs whose Latest Status is
Running.Job insights
In the job list, click Insights in the Actions column for a job to view a job overview, resource consumption details, the resource allocation of its quota group at a specific time, and trigger an intelligent job diagnosis.
Currently, intelligent job diagnosis supports only SQL jobs.
Job-level resource consumption data is available only for SQL, MapReduce, Spark, or Mars jobs that run for two minutes or more.
Job statistics chart
Based on your filter criteria, a stacked bar chart displays the number of jobs over time and by status, providing a high-level overview of job execution.
The time interval for each bar depends on the selected Time Range:
If the Time Range is within 24 hours: Each bar represents a minimum interval of 2 minutes. The number of bars is adaptive, up to a maximum of 24.
If the Time Range is greater than 24 hours and up to 48 hours: Each bar represents a 2-hour interval. The number of bars is adaptive, up to a maximum of 24.
If the Time Range is greater than 48 hours and up to 7 days: Each bar represents a 6-hour interval. The number of bars is adaptive, up to a maximum of 29.
The job statuses included in the statistics are:
Running: The job snapshot has a status of Running.
Finished: The job's status is Succeeded, Failed, or Terminated.
Because job snapshot data is collected every three minutes, some jobs may not be captured. As a result, the snapshot status for a job can be empty, even if the job is running.
You can click and drag within the chart to select and zoom in on a narrower time range.
Job list
The job list displays jobs that match your filter criteria and provides key information about them. MaxCompute offers two list views for different scenarios: a Regular Job List and a Snapshot List.
Regular Job List: Displays information for all jobs that ran within a specified time period.
Snapshot List: Displays a snapshot of jobs that were running at a specific point in time. It includes the snapshot status, as well as the CPU and memory usage, number of requests, and their percentage of the total at the time of the snapshot.
For a detailed explanation of the parameters displayed in the job list, see Job parameter description.
Limitations
The following job information is not currently collected:
Job snapshot data is collected every three minutes. Therefore, jobs that start and finish within a single three-minute collection interval are not captured in snapshots.
Data for some MaxCompute jobs initiated via PAI, especially those from RAM users, cannot be collected.
Data from jobs in projects on the Developer Edition (to be discontinued) cannot be collected.
Due to data processing latency, when you view the current job list, a job may show a current status of Running even if it has already finished in LogView. This is more common for very short jobs. Always consider the status in LogView to be definitive.
Parameters
Global job filter parameters
These parameters affect the job statistics chart and the job list.
Parameter | Description |
Time Range |
|
Choose Project | Filters jobs by MaxCompute project name. You can select multiple projects. By default, this field is empty. |
Select Quota |
|
Job Type |
|
Job list search parameters
Use these parameters to refine the results in the job list. These parameters only affect the job list.
Parameter | Description |
Instance ID |
|
Job Owner |
|
ExtNodeId |
|
ExtNodeName | Searches by the job name from the originating system of the MaxCompute job. |
Signature | Searches by the signature of an SQL job. This parameter applies only to SQL jobs. Use this signature to find all instances from each execution of the same SQL query. By default, this field is empty. |
Job list column filter parameters
Use these parameters to further filter the job list. They only affect the job list. To apply a filter, click the filter icon 
next to a column name.
Parameter | Description |
Latest Status |
|
Priority |
|
Intelligent Diagnostics |
|
General list columns
Column name | Description |
Instance ID | The unique instance ID for a MaxCompute job. This column also displays the project, quota group, and job type.
|
Latest Status | The most recent status of the job. |
Job Owner | The Alibaba Cloud account that ran the MaxCompute job. You can use this information to identify the person responsible for a job. If a job is consuming excessive resources and affecting other tasks, you can contact the job owner to stop the job. For instructions on how to stop a job, see Instance operations. |
Priority | The priority of the job, which ranges from 0 to 9. A smaller number indicates a higher priority. Higher-priority jobs acquire compute resources before lower-priority ones. For more information, see Job priority. |
Submission Time | The time when the instance was submitted. |
Start Running Time | The time when the job received the first computing resource. If the running time of a job is too short or a job does not consume computing resources, such as DDL statements, use the submission time of the job instead. This column is hidden by default. You can display it by customizing the column settings. |
Waiting Duration | The duration from the time a job is submitted to the time it starts to run. If the job has no start time, the duration from the submission time to the snapshot time is displayed. This column is hidden by default. You can display it by customizing the column settings. |
Execution Duration | The duration from the start time to the end time of a job. This column is hidden by default. You can display it by customizing the column settings. |
End Time | The end time of a job. |
Total Duration | The total duration from the time a job is submitted to the time the job is terminated. |
Total Used CPU Resources | The total CPU resources consumed by the job throughout its execution. The unit is |
Total Amount of Used Memory Resources | The total memory resources consumed by the job throughout its execution. The unit is |
CPU Utilization Percentage Snapshot | The latest CPU usage percentage collected for the job 1 minute before the deadline of the time range. It is calculated by using the formula: CPU Cores in Use/(CPU Lower Limit for Reserved CUs + Elastically Reserved Minimum CPU Utilization). This information is not available for pay-as-you-go jobs or jobs that cannot collect snapshot information. Sorting is not supported. If you need to sort data, go to the snapshot list. |
Memory Usage Percentage Snapshot | The latest memory usage percentage collected 1 minute before the job deadline of the time range. It is calculated by using the formula: Memory Size Occupied/(Minimum Reserved CU Memory + Minimum Elastically Reserved CU Memory). This information is not available for pay-as-you-go jobs or jobs that cannot collect snapshot information. Sorting is not supported. If you need to sort data, go to the snapshot list. |
Input Size | The amount of input data the job processed. |
Intelligent Diagnostics | The tags generated based on the results of the Intelligent Diagnosis feature. |
ExtPlatformId | The client that initiated the job, such as DataWorks. The initiator must pass in information when starting a job. For the required data conventions, see Job Universal Identifier Conventions. |
ExtNodeId | The task ID that corresponds to the job initiator, such as the ID of a DataWorks scheduling node. The initiator must provide information when submitting a job. For the conventions on this information, see General Job Identifier Conventions. |
ExtNodeName | The task name that corresponds to the job initiator, such as the name of a DataWorks scheduling node.. The initiator is required to pass information when starting a job. For the information conventions, see General Job Identifier Convention. |
ExtNodeOnDuty | The task owner that corresponds to the job initiator, such as the owner of a DataWorks scheduling node. The initiator must pass information when starting a job. For the information passing conventions, see Job General Identifier Conventions. |
Signature | The signature of an SQL job. You can use this signature to search for the instance on which the SQL job runs. |
Snapshot list columns
Column name | Description |
Instance ID | The unique instance ID for a MaxCompute job. This column also displays the project, quota group, and job type.
|
Snapshot Time | The time when snapshot information is collected for the job. |
Snapshot Status | The status of a job at the snapshot time. |
Job Owner | The Alibaba Cloud account that ran the MaxCompute job. You can use this information to identify the person responsible for a job. If a job is consuming excessive resources and affecting other tasks, you can contact the job owner to stop the job. For instructions on how to stop a job, see Instance operations. |
Priority | The priority of the job, which ranges from 0 to 9. A smaller number indicates a higher priority. Higher-priority jobs acquire compute resources before lower-priority ones. For more information, see Job priority. |
CPU Cores in Use | Number of CPU cores in use for a job at the snapshot time. |
Requested CPU Cores | The number of requested CPU cores of a job at the snapshot time. |
CPU Satisfaction Rate | Number of CPU cores in use/Number of requested CPU cores of a job at the snapshot time. |
CPU Utilization Percentage Snapshot | The latest CPU usage percentage collected for the job 1 minute before the deadline of the time range. It is calculated by using the formula: CPU Cores in Use/(CPU Lower Limit for Reserved CUs + Elastically Reserved Minimum CPU Utilization). This information is not available for pay-as-you-go jobs or jobs that cannot collect snapshot information. Sorting is not supported. If you need to sort data, go to the snapshot list. |
Memory Size Occupied | The size of requested memory space of a job at the snapshot time. The unit is displayed adaptively. |
Requested Memory Size | The size of requested memory space of a job at the snapshot time. The unit is displayed adaptively. |
Memory Satisfaction Rate | Memory size occupied/Requested memory size of a job at the snapshot time. |
Memory Usage Percentage Snapshot | The memory usage percentage of a job at the observation time, which is calculated by using the following formula: Amount of used memory resources/(Amount of reserved memory resources + Amount of elastically reserved memory resources). This metric is unavailable for pay-as-you-go jobs or if a snapshot was not captured. |
Submission Time | The time when a job was submitted. |
Total Duration | The total duration from the time a job is submitted to the time the job is terminated. |
ExtPlatformId | The client that initiated the job, such as DataWorks. The initiator must pass information when submitting a job. For the conventions on this information, see General Job Identifier Conventions. |
ExtNodeId | The task ID that corresponds to the job initiator, such as the ID of a DataWorks scheduling node. The client must pass information when it initiates a job. For the conventions on this information, see General Job Identifier Conventions. |
ExtNodeOnDuty | The task owner that corresponds to the job initiator, such as the owner of a DataWorks scheduling node. The initiator must pass information when initiating a job. For the information passing conventions, see General Job Identifier Convention. |
Signature | The signature of an SQL job. You can use this signature to search for the instance on which the SQL job runs. |
Common O&M scenarios
View run details of a specific job
Use case
This scenario covers viewing the execution status of a job initiated by a DataWorks hourly scheduling node or auditing a specific MaxCompute job.
Procedure
Log in to the MaxCompute console and select a region in the upper-left corner.
In the left-side navigation pane, choose .
On the Jobs page, select a Time Range and click Search.
Above the job list, select ExtNodeId or Instance ID and enter the corresponding value for the job.
Click the
icon to filter the job list again.In the query results, click LogView in the Actions column for an instance to view its detailed run information. For more information about LogView, see View job run information by using LogView 2.0.
View job run details by time period
Use case
This scenario covers viewing the jobs that were run for Project_1 and Project_2 over the last day, analyzing which jobs failed, and handling the failed jobs.
Procedure
Log in to the MaxCompute console and select a region in the upper-left corner.
In the left-side navigation pane, choose .
Set the Time Range to 1d, or set a custom Time Range from
00:00:00on the current day to the current time.From the Choose Project drop-down list, select Project_1 and Project_2.
In the query results, click LogView in the Actions column for an instance to view its detailed run information. For more information about LogView, see View job run information by using LogView 2.0.
View job resource usage on a subscription quota
Use case
When the resource usage of the Default Subscription Quota is high and many jobs are waiting, this scenario covers identifying the specific jobs that are consuming the quota resources.
Procedure
Log in to the MaxCompute console and select a region in the upper-left corner.
In the left-side navigation pane, choose .
Set the Time Range to 1h, or specify a custom start and end time.
Set the Select Quota parameter to
Default Subscription Quotaand click Search.In the list of query results, check the CPU Utilization Percentage Snapshot and Memory Usage Percentage Snapshot for jobs whose Latest Status is
Running. Check whether the job that consumes the most resources meets business requirements and use other job information to determine whether the job is performing as expected or should be terminated.In the query results, click LogView in the Actions column for an instance to view its detailed run information. For more information about LogView, see View job run information by using LogView 2.0.
View run details of Query Acceleration jobs
Use case
This scenario covers viewing the execution status and details of Query Acceleration jobs over the last day.
Procedure
Log in to the MaxCompute console and select a region in the upper-left corner.
In the left-side navigation pane, choose .
Set the Time Range to 1d and the Job Type to SQLRT(Query Acceleration), and then click Search.
In the query results, click LogView in the Actions column for an instance to view its detailed run information. For more information about LogView, see View job run information by using LogView 2.0.
For jobs that use the Query Acceleration feature, multiple SQL statements may be executed in the same session. A session corresponds to an Instance ID. You can use the LogView of the Instance ID to view the execution status of all SQL statements in the session. Therefore, take note of the following when you view Query Acceleration jobs on the Jobs page:
If the session has not exited, which means that some SQL statements have been completed and some are still running, the Latest Status of the job is
Running.If the session expires or is closed, the Latest Status of the job is
Cancelled.
Job resource consumption and computing quota allocation
Use case
If a job runs for a long period of time and the cause cannot be located in LogView, or if a job runs slower than expected, you need to determine if the issue is caused by insufficient resources.
Procedure
Log in to the MaxCompute console and select a region in the upper-left corner.
In the left-side navigation pane, choose .
Select a Time Range and filter by Select Quota. Then, click Search.
Find the target Instance ID and click Insights in the Actions column to go to the Job Insights page.
On the Resource Consumption tab, view the resource consumption of the job throughout its lifecycle.
The resource consumption chart shows the curves of used CUs and waiting CUs over time at the job level, and the trend of used CUs and waiting CUs over time at the quota level. If the job uses a small number of CUs but the quota to which the job belongs uses a large number of CUs or even continuously reaches the upper limit, the quota resources are insufficient. In this case, other jobs are competing for compute resources with the current job.
Click a time point on the horizontal axis of the resource consumption chart to view the resource allocation of the computing quota at that point in time, including the job quantity and priority distribution. Click the color block that corresponds to a priority to go to the job list and view the detailed information about the job. This helps you identify the jobs that are competing for compute resources with the current job. You can optimize task execution based on your business requirements by adjusting the job priority or managing compute resources. For more information, see job priority or Compute resources - Quota management.
Next steps
If a job consistently consumes a high percentage of resources while other jobs are queued, consider the following actions:
If the job does not match your business needs, terminate the job.
If the job matches your business needs, the resource configuration for the quota may be suboptimal. Optimize your resource configuration. For more information, see Compute resource configuration optimization.
Related documents
For the commands to view job information and status, and to stop jobs, see instance operations.