Schedule Spark jobs with Azkaban - AnalyticDB - Alibaba Cloud Documentation Center

Prerequisites

An AnalyticDB for MySQL Enterprise Edition, Basic Edition, or Data Lakehouse Edition cluster is created.
You have created a job resource group or an interactive resource group for the Spark engine in your AnalyticDB for MySQL cluster.
Azkaban is installed.
Beeline is installed.
You have added the IP address of the server that runs Azkaban to the IP address whitelist of your AnalyticDB for MySQL cluster.

Schedule Spark SQL jobs

You can use Spark-Submit to submit Spark SQL jobs and Spark jobs to AnalyticDB for MySQL.

Batch

Install the Spark-Submit command-line tool and configure the related parameters.

Create a workflow file and compress its parent folder into a ZIP file.

nodes:
  - name: SparkPi
    type: command
    config:
      command: /<your path>/adb-spark-toolkit-submit/bin/spark-submit 
                --class com.aliyun.adb.spark.sql.OfflineSqlTemplate 
                local:///opt/spark/jars/offline-sql.jar 
                "show databases" 
                "select 100"
    dependsOn:
      - jobA
      - jobB

  - name: jobA
    type: command
    config:
      command: echo "This is an echoed text."

  - name: jobB
    type: command
    config:
      command: pwd

Important

Replace <your path> with the actual installation path of the Spark-Submit command-line tool.
Do not use a line continuation character (\) in the command for the command node.

Create a project and upload the ZIP file from step 2.
1. Open the Azkaban web interface. In the top navigation bar, click Project.
2. In the upper-right corner of the page, click Create Project.
3. In the Create Project dialog box, set the Name and Description parameters.
4. In the upper-right corner of the page, click Upload.
5. In the Upload Project Files dialog box, upload the ZIP file and click Upload.
Run the workflow.
1. On the Project page, click the Flows tab.
2. Click Execute Flow.
3. Click Execute.
4. In the Flow submitted dialog box, click Continue.
View the workflow details.
1. In the top navigation bar, click Executing.
2. Click the Recently Finished tab.
3. Click an Execution ID and then click the Job List tab to view the execution details for each job.
4. Click Logs to view job logs.

Interactive

Obtain the Spark Interactive resource group endpoint.
1. Log on to the AnalyticDB for MySQL console. In the upper-left corner of the console, select a region. In the left-side navigation pane, click Clusters. Find the cluster that you want to manage and click the cluster ID.
2. In the navigation pane on the left, choose Cluster Management > Resource Management, and then click the Resource Groups tab.
3. Find the resource group and click Details in the Actions column to view the internal endpoint and public endpoint. You can click the icon next to an endpoint to copy it, or click the icon within the parentheses of the Port number to copy the JDBC connection string.
  
  In the following cases, you must click Apply for Endpoint next to Public Address to manually apply for a public endpoint.
  - The client tool used to submit Spark SQL jobs is deployed on your local machine or an external server.
  - The client tool used to submit Spark SQL jobs is deployed on an ECS instance, and the ECS instance and the AnalyticDB for MySQL cluster are not in the same VPC.
  The connection information also includes fields such as the public and VPC port (default: 10000), VPC ID, VSwitch ID, driver class (org.apache.hive.jdbc.HiveDriver), and driver download URL.

Create a workflow file and compress its parent folder into a ZIP file.

nodes:
  - name: jobB
    type: command
    config:
      command: <path> -u "jdbc:hive2://amv-t4n83e67n7b****sparkwho.ads.aliyuncs.com:10000/adb_demo" -n spark_interactive_prod/spark_user -p "spark_password" -e "show databases;show tables;"

    dependsOn:
      - jobA

  - name: jobA
    type: command
    config:
      command: <path> -u "jdbc:hive2://amv-t4n83e67n7b****sparkwho.ads.aliyuncs.com:10000/adb_demo" -n spark_interactive_prod/spark_user -p "spark_password" -e "show tables;"

Parameters:

Parameter	Description
path	The path to the Beeline client. Example: `/path/to/spark/bin/beeline`.
-u	Enter the connection address obtained in Step 1. Replace `default` in the connection address with the actual database name, and remove `resource_group=<resource group name>` from the connection address. Example: `jdbc:hive2://amv-t4naxpqk****sparkwho.ads.aliyuncs.com:10000/adb_demo`.
-n	The AnalyticDB for MySQL database account and resource group name, in the format `resource_group_name/database_account_name`. For example, if the resource group name is spark_interactive_prod and the database account name is spark_user, enter `spark_interactive_prod/spark_user`.
-p	The password for the AnalyticDB for MySQL database account.
-e	The SQL statements to run. Use a semicolon (;) to separate multiple statements.

Create a project and upload the ZIP file from the previous step.
1. Open the Azkaban web interface. In the top navigation bar, click Project.
2. In the upper-right corner of the page, click Create Project.
3. In the Create Project dialog box, set the Name and Description parameters.
4. In the upper-right corner of the page, click Upload.
5. In the Upload Project Files dialog box, upload the ZIP file and click Upload.
Run the workflow.
1. On the Project page, click the Flows tab.
2. Click Execute Flow.
3. Click Execute.
4. In the Flow submitted dialog box, click Continue.
View the workflow details.
1. In the top navigation bar, click Executing.
2. Click the Recently Finished tab.
3. Click an Execution ID and then click the Job List tab to view the execution details for each job.
4. Click Logs to view job logs.

Schedule Spark JAR jobs

Install the Spark-Submit command-line tool and configure its parameters.

Note
Configure only the following required parameters: keyId, secretId, regionId, clusterId, and rgName. If your Spark JAR package is stored on your local machine, also configure Object Storage Service (OSS) parameters, such as ossUploadPath.

Create a workflow file and compress its parent folder into a ZIP file.

nodes:
  - name: SparkPi
    type: command
    config:
      command: /<your path>/adb-spark-toolkit-submit/bin/spark-submit 
                --class org.apache.spark.examples.SparkPi 
                --name SparkPi 
                --conf spark.driver.resourceSpec=medium 
                --conf spark.executor.instances=2 
                --conf spark.executor.resourceSpec=medium 
                local:///tmp/spark-examples.jar 1000
    dependsOn:
      - jobA
      - jobB

  - name: jobA
    type: command
    config:
      command: echo "This is an echoed text."

  - name: jobB
    type: command
    config:
      command: pwd

Important

Replace <your path> with the actual installation path of the Spark-Submit command-line tool.
Do not use a line continuation character (\) in the command for the command node.

Create a project and upload the ZIP file from step 2.
1. Open the Azkaban web interface. In the top navigation bar, click Project.
2. In the upper-right corner of the page, click Create Project.
3. In the Create Project dialog box, set the Name and Description parameters.
4. In the upper-right corner of the page, click Upload.
5. In the Upload Project Files dialog box, upload the ZIP file and click Upload.
Run the workflow.
1. On the Project page, click the Flows tab.
2. Click Execute Flow.
3. Click Execute.
4. In the Flow submitted dialog box, click Continue.
View the workflow details.
1. In the top navigation bar, click Executing.
2. Click the Recently Finished tab.
3. Click an Execution ID and then click the Job List tab to view the execution details for each job.
4. Click Logs to view job logs.