AnalyticDB:Spark editor

Prerequisites

Before you begin, ensure that you have:

• An AnalyticDB for MySQL Data Lakehouse Edition clusterData Lakehouse Edition

• A job resource group created for the cluster. See Data Lakehouse EditionCreate a resource group

• The required permissions granted to your Resource Access Management (RAM) user. See the "Grant permissions to a RAM user" section of Manage RAM users and permissions

• A database account for the cluster:

  • Alibaba Cloud account: a privileged account. See the "Create a privileged account" section of Create a database account

  • RAM user: a privileged account and a standard account, with the standard account associated with the RAM user. See Create a database account and Associate or disassociate a database account with or from a RAM user

• AnalyticDB for MySQL authorized to assume the AliyunADBSparkProcessingDataRole role. See Perform authorization

• A log storage path configured for Spark applications

To configure the log storage path: Log on to the AnalyticDB for MySQL console. Find the cluster and click its cluster ID. In the left-side navigation pane, choose Job Development > Spark JAR Development, then click Log Settings. Select the default path or enter a custom path. The custom path cannot be the root directory of OSS — it must include at least one folder level.

Create and run a Spark application

1. Log on to the AnalyticDB for MySQL console. In the upper-left corner, select a region. In the left-side navigation pane, click ClustersData Lakehouse Edition. On the Data Lakehouse Edition tab, find the cluster and click its cluster ID.

2. In the left-side navigation pane, choose Job Development > Spark JAR Development.

3. On the Spark JAR Development page, click the icon to the right of Applications.

4. In the Create Application panel, configure the following parameters.

   Parameter	Description
   Name	Name of the application or directory. File names are case-insensitive.
   Type	Application: creates a file-based Spark template. Directory: creates a folder to organize applications.
   Parent Level	The parent directory for the file or folder.
   Job Type	The type of Spark job: Batch for batch processing, Streaming for streaming applications, or SQL Engine for Spark distributed SQL engine workloads.

5. Click OK.

6. In the Spark editor, configure the application. See Overview for configuration details.

7. Before running the application, select a job resource group and an application type in the editor. Then choose one of the following actions:

   • Click Save to save the application for later use.

   • Click Run Now to run the application immediately. The status updates in real time on the Applications tab.

   Note

   By default, no retry is performed after a failure. To configure retry behavior, set the spark.adb.maxAttempts and spark.adb.attemptFailuresValidityInterval parameters before running. See Spark application configuration parameters for details.

Monitor a Spark application

On the Applications tab, search for an application by application ID. Use the following actions in the Actions column based on what you need to investigate.

To view retry history across all applications, click the Execution History tab.