All Products
Search

This Product

    AnalyticDB:Schedule Spark jobs with DolphinScheduler

    Document Center

    AnalyticDB:Schedule Spark jobs with DolphinScheduler

    Last Updated:Aug 21, 2026

    DolphinScheduler is a distributed and scalable open-source workflow scheduling system with a visual Directed Acyclic Graph (DAG) interface. It helps you efficiently run and manage big data processes. You can use the DolphinScheduler web UI to easily create, edit, and schedule Spark jobs for AnalyticDB for MySQL.

    Prerequisites

    Schedule Spark SQL jobs

    You can run Spark SQL jobs on AnalyticDB for MySQL in two modes: batch and interactive. The procedures are detailed in the following sections.

    Batch

    1. Install the spark-submit command-line tool and configure its parameters.

      Note

      You only need to configure the required parameters: keyId, secretId, regionId, clusterId, and rgName.

    2. Create a project.

      1. Log in to the DolphinScheduler web UI. In the top navigation bar, click Project.

      2. Click Create Project.

      3. In the Create Project dialog box, configure parameters such as Project Name and Owned Users.

    3. Create a workflow.

      1. Click the project name to go to the Process Definition page.

      2. Click Create Workflow to open the workflow DAG edit page.

      3. From the component list in the left pane, drag Shell to the canvas on the right.

      4. In the Current node settings dialog box, configure the following parameters:

        Parameter

        Description

        Node Name

        The name of the workflow node.

        Script

        The installation path of the spark-submit command-line tool and the job code for your Spark SQL job. Example: /root/adb-spark-toolkit-submit/bin/spark-submit --class com.aliyun.adb.spark.sql.OfflineSqlTemplate local:///opt/spark/jars/offline-sql.jar "show databases" "select 100"

        Important

        Specify the full installation path of spark-submit in the script. Otherwise, the scheduling task cannot find the command.
        Note

        For more information about other parameters, see DolphinScheduler Task Parameters.

      5. Click OK.

      6. In the upper-right corner of the page, click Save. In the Basic Information dialog box, configure parameters such as Workflow Name, and then click OK.

    4. Run the workflow.

      1. In the workflow's Actions column, click the image icon to publish it.

      2. In the workflow's Actions column, click the image icon.

      3. In the Configure Parameters dialog box, configure the required parameters.

      4. Click OK to run the workflow.

    5. View workflow details.

      1. In the navigation pane on the left, click Task Instances.

      2. In the Operation column, click the image icon to view the execution results and logs.

    Interactive

    1. Obtain the Spark Interactive resource group endpoint.

      1. Log on to the AnalyticDB for MySQL console. In the upper-left corner of the console, select a region. In the left-side navigation pane, click Clusters. Find the cluster that you want to manage and click the cluster ID.

      2. In the navigation pane on the left, choose Cluster Management > Resource Management, and then click the Resource Groups tab.

      3. Find the resource group and click Details in the Actions column to view the internal endpoint and public endpoint. You can click the image icon next to an endpoint to copy it, or click the image icon within the parentheses of the Port number to copy the JDBC connection string.

        In the following cases, you must click Apply for Endpoint next to Public Address to manually apply for a public endpoint.

        • The client tool used to submit Spark SQL jobs is deployed on your local machine or an external server.

        • The client tool used to submit Spark SQL jobs is deployed on an ECS instance, and the ECS instance and the AnalyticDB for MySQL cluster are not in the same VPC.

        The connection information also includes fields such as the public and VPC port (default: 10000), VPC ID, VSwitch ID, driver class (org.apache.hive.jdbc.HiveDriver), and driver download URL.

    2. Create a data source.

      1. Log in to the DolphinScheduler web UI. In the top navigation bar, click Datasource.

      2. Click Add Data Source and select Spark as the data source type.

      3. In the Add Data Source dialog box, configure the following parameters:

        Parameter

        Description

        Data Source

        Select Spark.

        Datasource name

        The name of the data source.

        IPHost Name

        Enter the connection URL from Step 1. In the URL, replace default with your actual database name and remove the resource_group=<resource_group_name> suffix.

        Example: jdbc:hive2://amv-t4naxpqk****sparkwho.ads.aliyuncs.com:10000/adb_demo.

        Port

        The port number of the Spark interactive resource group is fixed at 10000.

        Username

        The database account for AnalyticDB for MySQL.

        Database Name

        The database name of the AnalyticDB for MySQL cluster.
        Note

        Other parameters are optional. For more information, see MySQL Datasource.

      4. Click Test Connection. After the test succeeds, click OK.

    3. Create a project.

      1. Log in to the DolphinScheduler web UI. In the top navigation bar, click Project.

      2. Click Create Project.

      3. In the Create Project dialog box, configure parameters such as Project Name and Owned Users.

    4. Create a workflow.

      1. Click the project name to go to the Process Definition page.

      2. Click Create Workflow to open the workflow DAG edit page.

      3. From the component list in the left pane, drag SQL to the canvas on the right.

      4. In the Current node settings dialog box, configure the following parameters:

        Parameter

        Description

        Data Source Type

        Select Spark.

        Datasource instance

        Select the data source created in Step 2.

        SQLType

        The type of the SQL job, such as Queries or Non Query.

        SQL statement

        The SQL statement to run.

      5. Click OK.

      6. In the upper-right corner of the page, click Save. In the Basic Information dialog box, configure parameters such as Workflow Name, and then click OK.

    5. Run the workflow.

      1. In the workflow's Actions column, click the image icon to publish it.

      2. In the workflow's Actions column, click the image icon.

      3. In the Configure Parameters dialog box, configure the required parameters.

      4. Click OK to run the workflow.

    6. View workflow details.

      1. In the navigation pane on the left, click Task Instances.

      2. In the Operation column, click the image icon to view the execution results and logs.

    Schedule Spark JAR jobs

    1. Install the spark-submit command-line tool and configure its parameters.

      Note

      You only need to configure the required parameters: keyId, secretId, regionId, clusterId, and rgName. If your Spark JAR package is stored locally, you must also configure Object Storage Service (OSS)-related parameters, such as ossUploadPath.

    2. Create a project.

      1. Log in to the DolphinScheduler web UI. In the top navigation bar, click Project.

      2. Click Create Project.

      3. In the Create Project dialog box, configure parameters such as Project Name and Owned Users.

    3. Create a workflow.

      1. Click the project name to go to the Process Definition page.

      2. Click Create Workflow to open the workflow DAG edit page.

      3. From the component list in the left pane, drag Shell to the canvas on the right.

      4. In the Current node settings dialog box, configure the following parameters:

        Parameter

        Description

        Node Name

        The name of the workflow node.

        Script

        The installation path of the spark-submit command-line tool and the job code for your Spark job. Example:

        /root/adb-spark-toolkit-submit/bin/spark-submit --class org.apache.spark.examples.SparkPi --name SparkPi --conf spark.driver.resourceSpec=medium --conf spark.executor.instances=2 --conf spark.executor.resourceSpec=medium local:///tmp/spark-examples.jar 1000.

        Important

        Specify the full installation path of spark-submit in the script. Otherwise, the scheduling task cannot find the command.
        Note

        For more information about other parameters, see DolphinScheduler Task Parameters.

      5. Click OK.

      6. In the upper-right corner of the page, click Save. In the Basic Information dialog box, configure parameters such as Workflow Name, and then click OK.

    4. Run the workflow.

      1. In the workflow's Actions column, click the image icon to publish it.

      2. In the workflow's Actions column, click the image icon.

      3. In the Configure Parameters dialog box, configure the required parameters.

      4. Click OK to run the workflow.

    5. View workflow details.

      1. In the navigation pane on the left, click Task Instances.

      2. In the Operation column, click the image icon to view the execution results and logs.