Execute Spark SQL Batch Jobs with ExecuteSparkWarehouseBatchSQL - AnalyticDB for MySQL - Alibaba Cloud - AnalyticDB

Operation description

Public endpoint of a region: adb.<region-id>.aliyuncs.com. Example: adb.cn-hangzhou.aliyuncs.com.
VPC endpoint of a region: adb-vpc.<region-id>.aliyuncs.com. Example: adb-vpc.cn-hangzhou.aliyuncs.com.

Note

If you receive a 409 error when sending requests from the China (Qingdao), China (Shenzhen), China (Guangzhou), or China (Hong Kong) region, contact technical support.

Try it now

Try this API in OpenAPI Explorer, no manual signing needed. Successful calls auto-generate SDK code matching your parameters. Download it with built-in credential security for local usage.

Test

RAM authorization

The table below describes the authorization required to call this API. You can define it in a Resource Access Management (RAM) policy. The table's columns are detailed below:

Action: The actions can be used in the Action element of RAM permission policy statements to grant permissions to perform the operation.
API: The API that you can call to perform the action.
Access level: The predefined level of access granted for each API. Valid values: create, list, get, update, and delete.
Resource type: The type of the resource that supports authorization to perform the action. It indicates if the action supports resource-level permission. The specified resource must be compatible with the action. Otherwise, the policy will be ineffective.
- For APIs with resource-level permissions, required resource types are marked with an asterisk (*). Specify the corresponding Alibaba Cloud Resource Name (ARN) in the Resource element of the policy.
- For APIs without resource-level permissions, it is shown as All Resources. Use an asterisk (*) in the Resource element of the policy.
Condition key: The condition keys defined by the service. The key allows for granular control, applying to either actions alone or actions associated with specific resources. In addition to service-specific condition keys, Alibaba Cloud provides a set of common condition keys applicable across all RAM-supported services.
Dependent action: The dependent actions required to run the action. To complete the action, the RAM user or the RAM role must have the permissions to perform all dependent actions.

Action

Access level

Resource type

Condition key

Dependent action

adb:ExecuteSparkWarehouseBatchSQL

none

*All Resource

*

None

Request parameters

Parameter	Type	Required	Description	Example
Agency	string	No	The client name.	Dataworks.
DBClusterId	string	Yes	The cluster ID.	amv-bp11q28kvl688****
ResourceGroupName	string	Yes	The resource group name.	test
Query	string	Yes	The SQL statements to execute in batches. Separate multiple SQL statements with semicolons (;). The execution engine runs the statements sequentially within the same session.	CREATE TABLE user(id INT, name STRING); INSERT INTO t VALUE(1, 'Bob'); SELECT * FROM t;
Schema	string	No	The database name.	adb_demo
ExecuteResultLimit	integer	No	The maximum size of execution result data that can be written to Object Storage Service (OSS). Unit: MB. Default value: 4096. Because compressed object sizes are difficult to estimate, the actual data written to OSS is smaller than the specified value.	4096
ExecuteTimeLimitInSeconds	integer	No	The maximum execution duration. Unit: seconds. If SQL statements are not completed within the specified period after submission, they are marked as a timeout error. Default value: 360000 (equivalent to 100 hours).	3600000
RuntimeConfig	string	No	The additional runtime parameters in JSON format.	{ "OSSURL": "oss://testBucketname/" }

Response elements

Element	Type	Description	Example
	object	The response body.
RequestId	string	The request ID.	CBE843D8-964D-5EA3-9D31-822125611B6E
Data	SparkBatchSQL	The returned data.	true

Examples

Success response

JSON format

{
  "RequestId": "CBE843D8-964D-5EA3-9D31-822125611B6E",
  "Data": {
    "QueryId": "sq172352176****",
    "QueryState": "RUNNING",
    "QuerySubmissionTime": 1723521767000,
    "QueryStartTime": 1723521767000,
    "QueryEndTime": 1723521767000,
    "Schema": "default",
    "Uid": 0,
    "DBClusterId": "amv-bp17s73f016****",
    "ResourceGroupName": "spark_rg",
    "Query": "SELECT * FROM db.tb",
    "Statements": [
      {
        "ResultUri": "oss://yourbucket/result.json",
        "EndTime": 1723521767000,
        "AppId": "s20240122jsdx****",
        "StatementId": "sq1723521767****-0001",
        "State": "RUNNING",
        "StartTime": 1723521767000,
        "Error": "table is not found",
        "Code": "SELECT 100",
        "Result": "+---+-----+\n|age| name|\n+---+-----+\n| 14|  Tom|\n| 23|Alice|\n+---+-----+"
      }
    ],
    "ErrMessage": "schema is not found"
  }
}

Error codes

HTTP status code	Error code	Error message	Description
400	Spark.InvalidParameter	Invalid parameter value: %s	Incorrect input parameter:%s.
400	Spark.InvalidState	The object of the operation is in an invalid state: %s	The operation object is invalid.
500	Spark.ServerError	The Spark control component system encountered an error, please create a ticket to solve the problem or concat the supported engineer on duty. Error message: %s	An error occurred on the Spark control component system. Submit a ticket or contact technical support.
404	Spark.ObjectNotFound	The object is not found. More information: %s

See Error Codes for a complete list.

Release notes

See Release Notes for a complete list.