Query Spark Job Run History with ListJobRuns API - E-MapReduce - Alibaba Cloud - E-MapReduce

Queries a list of Spark jobs.

Try it now

Try this API in OpenAPI Explorer, no manual signing needed. Successful calls auto-generate SDK code matching your parameters. Download it with built-in credential security for local usage.

Test

RAM authorization

The table below describes the authorization required to call this API. You can define it in a Resource Access Management (RAM) policy. The table's columns are detailed below:

Action: The actions can be used in the Action element of RAM permission policy statements to grant permissions to perform the operation.
API: The API that you can call to perform the action.
Access level: The predefined level of access granted for each API. Valid values: create, list, get, update, and delete.
Resource type: The type of the resource that supports authorization to perform the action. It indicates if the action supports resource-level permission. The specified resource must be compatible with the action. Otherwise, the policy will be ineffective.
- For APIs with resource-level permissions, required resource types are marked with an asterisk (*). Specify the corresponding Alibaba Cloud Resource Name (ARN) in the Resource element of the policy.
- For APIs without resource-level permissions, it is shown as All Resources. Use an asterisk (*) in the Resource element of the policy.
Condition key: The condition keys defined by the service. The key allows for granular control, applying to either actions alone or actions associated with specific resources. In addition to service-specific condition keys, Alibaba Cloud provides a set of common condition keys applicable across all RAM-supported services.
Dependent action: The dependent actions required to run the action. To complete the action, the RAM user or the RAM role must have the permissions to perform all dependent actions.

Action

Access level

Resource type

Condition key

Dependent action

emr-serverless-spark:ListJobRuns

list

*All Resource

*

None

Request syntax

GET /api/v1/workspaces/{workspaceId}/jobRuns HTTP/1.1

Path Parameters

Parameter	Type	Required	Description	Example
workspaceId	string	Yes	The workspace ID.	w-d2d82aa09155****

Request parameters

Parameter	Type	Required	Description	Example
nextToken	string	No	The pagination token that marks the position from which to start reading.	DD6B1B2A-5837-5237-ABE4-FF0C89568980
maxResults	integer	No	The maximum number of records to retrieve in a single request. Maximum value: 100.	20
name	string	No	The job name.	emr-spark-demo-job
creator	string	No	The UID of the user who created the job.	150976534701****
jobRunId	string	No	The job ID.	j-xxx
tags	array<object>	No	The list of tags.
	object	No	The tag object.
key	string	No	The tag key.	tag_key
value	string	No	The tag value.	value
states	array	No	The job states.	["Running","Submitted"]
	string	No	The job state. Valid values: Submitted: The job has been submitted. Pending: The job is pending. Running: The job is running. Success: The job succeeded. Failed: The job failed. Cancelling: The job is being cancelled. Cancelled: The job has been cancelled. CancelFailed: The job cancellation failed.	Running
startTime	object	No	The start time range of the job.
startTime	integer	No	The beginning of the start time range.	1709740800000
endTime	integer	No	The end of the start time range.	1710432000000
endTime	object	No	The end time range of the job.
startTime	integer	No	The beginning of the end time range.	1709740800000
endTime	integer	No	The end of the end time range.	1710432000000
resourceQueueId	string	No	The name of the resource queue on which the Spark job runs.	dev_queue
jobRunDeploymentId	string	No	The job ID of the streaming job deployment.	jd-b6d003f1930f****
regionId	string	No	The region ID.	cn-hangzhou
minDuration	integer	No	The minimum execution duration of the job, in milliseconds.	60000
isWorkflow	string	No	Specifies whether the job is a workflow job.	false
runtimeConfigs	string	No	The runtime configurations.	[{\"key\":\"mainClass\",\"value\":\"yourClass\"}]
applicationConfigs	string	No	The Spark configurations.	[{\"key\":\"spark.app.name\",\"value\":\"test\"}]

Response elements

Element	Type	Description	Example
	object	The response parameters.
jobRuns	array<object>	The list of Spark jobs.
	array<object>	The Spark node task object.
workspaceId	string	The workspace ID.	w-d2d82aa09155****
jobRunId	string	The job ID.	jr-231231
name	string	The job name.	jobName
state	string	The job states.	Running
stateChangeReason	object	The reason for the state change.
code	string	The error code.	0
message	string	The error message.	connection refused
submitTime	integer	The job submission time.	1684119314000
endTime	integer	The job end time.	1684119314000
codeType	string	The job code type. Valid values: SQL JAR PYTHON.	SQL
webUI	string	The job Web UI.	http://spark-ui
executionTimeoutSeconds	integer	The execution timeout period, in seconds.	3600
creator	string	The UID of the user who created the job.	150978934701****
tags	array	The tags.
	Tag	The job tag.
log	RunLog	The path of the run log.
releaseVersion	string	The version of the Spark DPI engine used to run the job.	esr-3.0.0 (Spark 3.4.3, Scala 2.12, Native Runtime)
jobDriver	JobDriver	The Spark Driver information. This parameter is not returned by the List operation.
configurationOverrides	object	The Spark advanced configurations. This parameter is not returned by the List operation.
configurations	array	A list of Spark configurations.
	Configuration	A Spark configuration object.
displayReleaseVersion	string	The version of the Spark DPI engine used to run the job.	esr-3.0.0 (Spark 3.4.3, Scala 2.12)
fusion	boolean	Indicates whether the Fusion engine acceleration is enabled.	true
vcoreSeconds	integer	The total number of allocated vcores multiplied by the number of seconds the job has been running.	8236
mbSeconds	integer	The total amount of allocated memory multiplied by the number of seconds the job has been running.	33030784
cuHours	number	The number of compute units (CUs) consumed during the job run cycle. This value is an estimate. The actual value is subject to the bill.	2.059
resourceQueueId	string		dev_queue
priority	string		5
requestId	string	The request ID.	DD6B1B2A-5837-5237-ABE4-FF0C8944****
nextToken	string	The position from which the data was read.	1
maxResults	integer	The maximum number of records returned in this request.	20
totalCount	integer	The total number of records that match the request conditions.	200
aggregations	object
	string

Examples

Success response

JSON format

{
  "jobRuns": [
    {
      "workspaceId": "w-d2d82aa09155****",
      "jobRunId": "jr-231231",
      "name": "jobName",
      "state": "Running",
      "stateChangeReason": {
        "code": "0",
        "message": "connection refused\n"
      },
      "submitTime": 1684119314000,
      "endTime": 1684119314000,
      "codeType": "SQL",
      "webUI": "http://spark-ui",
      "executionTimeoutSeconds": 3600,
      "creator": "150978934701****",
      "tags": [
        {
          "key": "workflowId",
          "value": "wf-123test"
        }
      ],
      "log": {
        "driverStdOut": "oss://bucket/path/to/stdout",
        "driverStdError": "oss://bucket/path/to/stderr",
        "driverSyslog": "oss://bucket/path/to/syslog",
        "driverStartup": "oss://bucket/path/to/startup"
      },
      "releaseVersion": "esr-3.0.0 (Spark 3.4.3, Scala 2.12, Native Runtime)",
      "jobDriver": {
        "sparkSubmit": {
          "entryPoint": "oss://bucket/path/to/entrypoint.jar",
          "entryPointArguments": [
            "arg1"
          ],
          "sparkSubmitParameters": "--conf spark.app.name=test"
        }
      },
      "configurationOverrides": {
        "configurations": [
          {
            "configFileName": "common.conf",
            "configItemKey": "hive.metastore.type",
            "configItemValue": "USER_RDS"
          }
        ]
      },
      "displayReleaseVersion": "esr-3.0.0 (Spark 3.4.3, Scala 2.12)",
      "fusion": true,
      "vcoreSeconds": 8236,
      "mbSeconds": 33030784,
      "cuHours": 2.059,
      "resourceQueueId": "dev_queue",
      "priority": "5"
    }
  ],
  "requestId": "DD6B1B2A-5837-5237-ABE4-FF0C8944****",
  "nextToken": "1",
  "maxResults": 20,
  "totalCount": 200,
  "aggregations": {
    "key": ""
  }
}

Error codes

See Error Codes for a complete list.

Release notes

See Release Notes for a complete list.