AnalyticDB for MySQL:Perform job scheduling in Data Lakehouse Edition

Flowchart

Prerequisites

• If you use an Alibaba Cloud account to perform job scheduling, make sure that the following conditions are met:
  • A privileged account is created for an AnalyticDB for MySQL cluster. For more information, see Create a database account.
  • A job resource group with at least 8 AnalyticDB Compute Units (ACUs) of reserved computing resources is created. For more information, see Create a resource group.
• If you perform job scheduling as a Resource Access Management (RAM) user, make sure that the following conditions are met:
  • The AliyunServiceRoleForAnalyticDBForMySQL role of AnalyticDB for MySQL is configured. For more information, see Role description.
  • A standard account is created for an AnalyticDB for MySQL cluster and associated with the RAM user. For more information, see Associate or disassociate a database account with or from a RAM user.
  • A job resource group with at least 8 AnalyticDB Compute Units (ACUs) of reserved computing resources is created. For more information, see Create a resource group.

Project management

Project management allows you to create projects, edit and delete existing projects, and manage members.

Create a project

1. Log on to the AnalyticDB for MySQL console.
2. In the upper-left corner of the page, select a region.
3. In the left-side navigation pane, click Clusters.
4. On the Data Lakehouse Edition (V3.0) tab, find the cluster that you want to manage and click the Cluster ID.
5. In the left-side navigation pane, click Job Scheduling.
6. On the Projects tab, click Create Project.
7. In the panel that appears, enter a project name and click OK.

Manage members

1. Find a project and click the icon in the Actions column.
2. Click Add Member.
3. In the Members section, select unauthorized accounts and click the icon.
   Note Unauthorized accounts refer to all RAM users that are not added to member management within an Alibaba Cloud account.
4. Click OK.

Workflow orchestration

Create a workflow

1. In the upper-right corner, choose Workflows > Workflow Templates.
2. Click Create Workflow.
   Note SQL and Spark task nodes can be created.
3. On the Create Workflow page, drag a node to the canvas and configure its parameters.
   The following table describes the parameters configured for SQL task nodes.

   Parameter	Description
   Node Name	The name of the SQL task node.
   Schema	The schema configured for the SQL task node.
   SQL Statements	The SQL statements. You can enter SQL statements or select existing SQL statements. Existing SQL statements refer to the SQL statements saved on the page after you choose Job Editor > SQL Development.
   Custom Parameters	The custom parameters that can be used as variables in SQL statements. For example, you can set the prop of a custom parameter to customer and its value to ${table}. In this case, the SELECT * FROM ${table}; statement can be changed to SELECT * FROM customer;.
   Pre-SQL	The pre-SQL statements that are executed before SQL statements of the current workflow. Only DML statements are allowed, such as INSERT, UPDATE, and DELETE.
   Post-SQL	The post-SQL statements that are executed after SQL statements of the current workflow. Only DDL statements are allowed, such as CREATE and DROP.
   Retry Attempts	The maximum number of retry attempts for the current SQL statements and the interval between each attempt. Note If you set the maximum number of retries to 3 and the first retry succeeds, no further retries are attempted.
   Description	The description of the SQL task node.

   The following table describes the parameters configured for Spark task nodes.

   Parameter	Description
   Node Name	The name of the Spark task node.
   Resource Group	The resource group configured for the Spark task node. You can select only a job resource group because Spark tasks are resource-intensive. If no options are available, you must create a job resource group first. For more information, see Create a resource group.
   Type	Batch: batch application Streaming: streaming application SQL: Spark SQL application
   Spark Statements	The Spark statements. You can enter the default task parameter templates or select existing Spark templates. Existing Spark templates refer to the Spark templates saved on the page after you choose Job Editor > Spark JAR Development.
   Retry Attempts	The maximum number of retry attempts for the current Spark statements and the interval between each attempt. Note If you set the maximum number of retries to 3 and the first retry succeeds, no further retries are attempted.
   Description	The description of the Spark task node.

4. Click OK to add.
5. In the upper-right corner of the canvas, click the icon. In the panel that appears, enter a workflow name, and configure global parameters.

   Parameter	Description
   Set Global Parameters	The global parameters that can be used as variables in Spark statements for workflow nodes. For example, you can set the value of the global variable ${table} to test. In this case, the SELECT * FROM ${table}; statement can be changed to SELECT * FROM test;.

Publish a workflow

After you create a workflow, click the icon to publish the workflow.

Execute a workflow

Workflow state management

After you execute a workflow, you can view the state of the workflow and pause or terminate the workflow on the Workflow Instances page. You can also view the states of tasks on the Task Instances page. If the workflow fails, you can re-execute or recover the workflow.

View the state of a workflow instance

View the state of a task instance

Monitor workflows