Submit a Spark application by calling the SubmitSparkApp API - AnalyticDB - Alibaba Cloud - AnalyticDB

Submits a Spark application.

Operation description

Regional public endpoint: adb.<region-id>.aliyuncs.com. Example: adb.cn-hangzhou.aliyuncs.com.
Regional Virtual Private Cloud (VPC) endpoint: adb-vpc.<region-id>.aliyuncs.com. Example: adb-vpc.cn-hangzhou.aliyuncs.com.

Note

If HTTP status code 409 is returned when you call this operation in the China (Qingdao), China (Shenzhen), China (Guangzhou), or China (Hong Kong) region, contact technical support.

Try it now

Try this API in OpenAPI Explorer, no manual signing needed. Successful calls auto-generate SDK code matching your parameters. Download it with built-in credential security for local usage.

Test

RAM authorization

The table below describes the authorization required to call this API. You can define it in a Resource Access Management (RAM) policy. The table's columns are detailed below:

Action: The actions can be used in the Action element of RAM permission policy statements to grant permissions to perform the operation.
API: The API that you can call to perform the action.
Access level: The predefined level of access granted for each API. Valid values: create, list, get, update, and delete.
Resource type: The type of the resource that supports authorization to perform the action. It indicates if the action supports resource-level permission. The specified resource must be compatible with the action. Otherwise, the policy will be ineffective.
- For APIs with resource-level permissions, required resource types are marked with an asterisk (*). Specify the corresponding Alibaba Cloud Resource Name (ARN) in the Resource element of the policy.
- For APIs without resource-level permissions, it is shown as All Resources. Use an asterisk (*) in the Resource element of the policy.
Condition key: The condition keys defined by the service. The key allows for granular control, applying to either actions alone or actions associated with specific resources. In addition to service-specific condition keys, Alibaba Cloud provides a set of common condition keys applicable across all RAM-supported services.
Dependent action: The dependent actions required to run the action. To complete the action, the RAM user or the RAM role must have the permissions to perform all dependent actions.

Action

Access level

Resource type

Condition key

Dependent action

adb:SubmitSparkApp

create

*SparkApplication

acs:adb:{#regionId}:{#accountId}:dbcluster/{#DBClusterId}/resourcegroup/{#ResourceGroup}

None

Request parameters

Parameter	Type	Required	Description	Example
AppName	string	No	The name of the application. The value can be up to 64 characters in length.	TestApp
AppType	string	No	The type of the application. Valid values: SQL STREAMING BATCH (default)	SQL
Data	string	Yes	The data of the application template. Note For information about the application template configuration, see Spark application configuration guide.	conf spark.driver.resourceSpec=small; conf spark.executor.instances=1; conf spark.executor.resourceSpec=small; conf spark.app.name=TestApp;
AgentSource	string	No	The type of the client. The value can be up to 64 characters in length.	CONSOLE
AgentVersion	string	No	The version of the client. The value can be up to 64 characters in length.	1.091
DBClusterId	string	Yes	The ID of the Enterprise Edition, Basic Edition, or Data Lakehouse Edition cluster. Note You can call the DescribeDBClusters operation to query the IDs of all AnalyticDB for MySQL clusters within a region.	amv-bp11q28kvl688****
ResourceGroupName	string	Yes	The name of the job resource group. Note You can call the DescribeDBResourceGroup operation to query the name of a resource group within a cluster.	adb
TemplateFileId	integer	No	The ID of the application template. Note You can call the GetSparkTemplateFullTree operation to query the application template ID.	15

Response elements

Element	Type	Description	Example
	object	The response parameters.
RequestId	string	The request ID.	1AD222E9-E606-4A42-BF6D-8A4442913CEF
Data	object	The returned data.
AppId	string	The application ID.	s202204132018hzprec1ac61a000****
AppName	string	The name of the application.	TestApp
Message	string	The alert message returned for the operation, such as task execution failure or insufficient resources. If no alert occurs, null is returned.	Insufficient resources.
State	string	The execution state of the application. Valid values: SUBMITTED STARTING RUNNING FAILING FAILED KILLING KILLED SUCCEEDING COMPLETED FATAL UNKNOWN	SUBMITTED

Examples

Success response

JSON format

{
  "RequestId": "1AD222E9-E606-4A42-BF6D-8A4442913CEF",
  "Data": {
    "AppId": "s202204132018hzprec1ac61a000****",
    "AppName": "TestApp",
    "Message": "Insufficient resources.",
    "State": "SUBMITTED"
  }
}

Error codes

HTTP status code	Error code	Error message	Description
400	Spark.App.ExceedQuotaLimitation	The requested resource exceeds the maximum limit: %s
400	Spark.App.InvalidAppTypeWhenSubmit	The specified AppType is INVALID or NULL. Please refer to the documentation to correct the parameters for %s.	The AppType type is illegal, please refer to using the document to specify the AppType suitable for your business.
400	Spark.App.InvalidResourceSpec	The requested resource type is not supported:\n %s
400	Spark.App.ParameterConflict	Conflicting parameters submitted:\n %s
400	Spark.App.ResourceNegativeError	The requested resource MUST be positive: %s
400	Spark.Config.InvalidAppFilePath	The main driver file MUST be a valid OSS path. Current is %s.
400	Spark.Config.invalidConnectors	The spark.adb.connectors configuration is invalid: %s
400	Spark.Config.InvalidDiskConfig	The requested disk mount parameter is invalid: %s
400	Spark.Config.InvalidHostAliasConfig	The requested host alias parameter %s is invalid. Example: 127.0.0.1 localhost local
400	Spark.Config.InvalidLogOssPath	The OSS address for log storage is illegal: %s
400	Spark.Config.InvalidRoleArnFormat	Configure RoleARN %s invalid format. It should match acs:ram::uid_number:role/role_name
400	Spark.Config.InvalidTimeZone	Unable to parse time zone configuration %s : %s
400	Spark.Config.MainAppFileNotExists	The main driver file is missing, [file] or [sqls] must be configured.
400	Spark.Config.OSSPathAccessError	The configured OSS address %s cannot be accessed.
400	Spark.Config.OSSPathNotExists	The OSS address %s requested does not exist or the permissions are insufficient.
400	Spark.Config.RoleArnVerifyFailed	RoleARN parameter verification failed. Error msg: %s when verify RoleArn %s
400	Spark.Config.SecurityGroupNotFound	The security group in the configuration does not exist or cannot be accessed. %s.	The configured resource group does not exist or cannot be accessed due to insufficient permissions. %s.
400	Spark.Config.VswitchNotFound	The vswitch in the configuration does not exist or cannot be accessed. %s.
400	Spark.InvalidParameter	Invalid parameter value: %s	Incorrect input parameter:%s.
400	Spark.InvalidState	The object of the operation is in an invalid state: %s	The operation object is invalid.
400	Spark.RoleArn.Invalid	%s is not found, or the RAM role has not been authorized.
400	Spark.SQL.BlankError	Input sql can not be blank string.
400	Spark.SQL.MultipleSQLError	Element in field [sqls] can not contain more than one sql statement: %s.
400	Spark.SQL.NotFoundExecutableSQLError	No executable statements are submitted. Please check the input SQL.
400	Spark.SQL.ParserError	Submit spark app failed when parser SQL %s. Error message: %s.
400	Spark.RoleARN.invalid	Current account fails to verify the permissions of the RAM system, please check the help information and configure the appropriate permissions for the current account.	Please make sure your current RAM user (ram_user_id %s) has been granted the system policy 'AliyunADBDeveloperAccess'. And then follow the section 'Perform authorization within an Alibaba Cloud account". This will automatically initialize the system role named 'aliyunadbsparkprocessingdatarole' in your cloud account. More details: https://www.alibabacloud.com/help/en/analyticdb/analyticdb-for-mysql/user-guide/create-the-aliyunadbsparkprocessingdatarole-role-for-a-ram-user-and-grant-permissions-to-the-role
500	Spark.ServerError	The Spark control component system encountered an error, please create a ticket to solve the problem or concat the supported engineer on duty. Error message: %s	An error occurred on the Spark control component system. Submit a ticket or contact technical support.
403	Spark.Forbidden	No permissions to access the resources: %s	Insufficient permissions to access the related resources. Information that you want to access: %s.
404	Spark.App.ContentNotFound	The requested content %s of the Spark application is not found.
404	Spark.App.NotFound	The Spark application %s is not found.
404	Spark.ObjectNotFound	The object is not found. More information: %s

See Error Codes for a complete list.

Release notes

See Release Notes for a complete list.