E-MapReduce:Manage workflow definitions

Prerequisites

A project is created. For more information, see Create a project.

Create a workflow

1. Go to the Project tab.

   1. Log on to the EMR console with your Alibaba Cloud account (main account) or a RAM user.

   2. In the navigation pane on the left, choose EMR Studio > Workflow.

   3. On the Workflow page, find the target workspace and click Console in the Actions column.

   4. On the Project Management page, click the name of an existing project.

2. In the navigation pane on the left, choose Data Development > Workflow Definition.

3. On the Workflow Definition page, click Create Workflow.

4. On the Create Workflow page, drag a node to the canvas. In the dialog box that appears, configure the parameters and click OK.

   This example uses a SHELL node. For more information about other node types, see Node types.

   Parameter	Description
   Node Name	The name of the node. A node name is unique in a workflow.
   Run Task	Normal (default): The task is executed when the workflow runs. No: The task is not executed when the workflow runs.
   Description	The feature description of the node.
   Task Priority	The priority of the node in the workflow. Default value: MEDIUM. Valid values: HIGHEST HIGH MEDIUM LOW LOWEST
   Retries After Failure	The maximum number of times that the system automatically retries to run the node if the workflow fails.
   Retry Interval upon Failure	The interval between two consecutive retries. Unit: minutes.
   Time for Delayed Execution	The amount of time that is delayed before the node is run. The default value is 0, which indicates that the node is immediately run after it is created. The node is delayed to be run only if you specify a value that is greater than 0 for this parameter. Unit: minutes.
   Timeout Alert	This feature is disabled by default. Turn on the Timeout Alert switch to set a timeout duration and select at least one timeout strategy. If you select the Timeout Alert strategy, the system sends a notification to the specified alert group when the task's runtime exceeds the specified duration. If you select the Task Failed Upon Timeout strategy, the task instance immediately fails when the task's runtime exceeds the specified duration.
   Node to Run Tasks	Resource Group for Scheduling (default): Submits the task to a node in the workflow's scheduling resource group. Worker Node in EMR Cluster (Submitted to YARN): The task is submitted on a worker node of the EMR cluster and occupies a YARN container. Warning This mode directly affects the file system and runtime environment of the EMR cluster. Proceed with caution. In this mode, you can also configure the following parameters: Queues for Scheduling: The YARN queue to which the task is submitted. Task Execution Container Memory (MB): The memory size of the task execution container. Default value: 1024. Number of vCPUs: The number of virtual cores (vCores) for the task execution container. Default value: 1. Priority: The priority of the task execution container, which ranges from 0 to 100. Default value: 0. A larger value indicates a higher priority. Execution User: The default user for task submission is dolphinscheduler. For a standard cluster (not a high-security cluster), if you need to specify an execution user, make sure that the YARN service includes the following configurations: yarn.nodemanager.container-executor.class=org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users=false Note For YARN-based tasks such as Spark, Hive, or Sqoop, set the Node to Run Tasks to Worker Node in EMR Cluster (Submitted to YARN). This allows you to fully leverage YARN's resource scheduling and management capabilities.
   Script	The custom Shell program. Note When you perform write operations in the script, note the following: You can write data to remote storage systems, such as distributed file systems like Alibaba Cloud OSS and HDFS. Do not write data to the local cluster file system. Avoid direct operations on local paths, such as /tmp/local_file.txt, in your script.
   Resources	Select the resource files that the task needs at runtime. Ensure you have created or uploaded the required files on the Resource Center > File Management page. You can then reference them in the task script by using their full paths, such as path/to/test.txt.
   Custom Parameters	Task-level custom parameters that replace ${variable} placeholders in the script. For more information, see Parameter reference.
   Pre-task	The ancestor node of the node.

5. Optional. Configure dependencies between nodes.

   EMR Workflow allows you to configure custom node dependencies between workflows.

   • Click the icon on the right side of a node and drag the connection line to connect it to another node.

   • Select a connection line or a node and click the icon in the upper-right corner to delete the dependency or the node.

6. Save the workflow.

   1. In the upper-right corner, click Save.

   2. In the dialog box that appears, configure the parameters and click OK.

      Parameter	Description
      Name	The name of the workflow.
      Description	The feature description of the workflow.
      Timeout Alert	This switch is turned off by default. If you turn on the Timeout Alert switch, you must specify a Timeout Period. An alert is sent if the runtime exceeds this timeout period.
      Execution Policy	The mode in which the instances of the workflow are run. Valid values: Parallel (Default): Workflow instances are run in parallel. Serial Waiting: Runs workflow instances sequentially. Each new instance waits for the previous one to complete before starting.
      Global Variables	A global variable is valid for all nodes of the workflow.

Operations on a workflow

On the Workflow Definition page, you can perform the following operations.

Run a workflow

Running a workflow generates a new instance on the Workflow Instance page.

1. In the Actions column, click Run.

2. In the dialog box that appears, configure the parameters and click OK.

   Parameter	Description
   Policy upon Failure	The policy for handling node failures in a workflow. Proceed: If a node fails, other nodes continue to run as scheduled. Finished (Default): If a node fails, its downstream nodes are terminated.
   Notification Policy	Sends an email notification based on the final status of the workflow run. The following options are available: Do Not Send (Default), Send for Successful Workflows, Send for Failed Workflows, and Always Send.
   Workflow Priority	The priority of the node in the workflow. Default value: MEDIUM. Valid values: HIGHEST HIGH MEDIUM LOW LOWEST
   Resource Groups for Scheduling	The resource group used for scheduling and running the task. Select an existing scheduling resource group from the drop-down list.
   Cluster/Template	The cluster or cluster template for execution. Available options are filtered based on the selected scheduling resource group. Note If you select a Cluster/Template for the Cluster Template, a new cluster is created based on the template when the workflow runs, and the cluster is released after the workflow is complete. If you have configured a Timeout Alert for the workflow definition, the cluster is also automatically released if the timeout rule is triggered. If you have not configured a Timeout Alert, the system automatically releases the cluster after it has been running for 24 hours.
   Backfill Data	Specifies whether to generate retroactive data based on the data backfill settings when the workflow is run within the specified time range. Note Backfilling rule: If the workflow has an active schedule, backfilling follows the schedule's frequency. Otherwise, the system generates runs once per day at 00:00 by default. Limitation: A single data backfilling operation can generate a maximum of 100 workflow instances. Select Yes and configure the following parameters: Dependency Mode: Supports Open and Off (default). This setting specifies whether data backfilling for an instance of the current workflow triggers data backfilling for its downstream dependent nodes. Retroactive data is generated for the workflows that depend on the current workflow only if the current workflow is in the Online state and scheduling settings are configured for the current workflow. Execution Method Serial Execution: Executes backfilling runs sequentially, creating one workflow instance for each interval in the specified time range. Parallel Execution (Default): Within the specified time range, backfilling runs for multiple days are executed concurrently, generating multiple workflow instances at the same time. For this mode, you must also configure the Customize Parallelism parameter, which specifies the maximum number of instances that can run in parallel. The maximum parallelism for each backfilling operation is 7. Note If the workflow definition uses the Parallel execution policy, you can only use Parallel Execution for data backfilling. Similarly, a workflow with a Serial Waiting policy requires Serial Execution for data backfilling. Scheduling Period: The date range for the workflow run.
   Global Parameters	A startup parameter and its value. The value is used to define a global variable or overwrite the existing value of the global variable when a new workflow instance is started.
   Dry Run	Specifies whether to perform a dry run for the workflow. If you perform a dry run for the workflow, a success log is recorded.

3. In the navigation pane on the left, choose Instance O&M>Workflow Instance to view the run status of the instance.

Import a workflow

1. On the Data Development > Workflow Definition page, click Import.

2. Click Upload, select the exported workflow definition file (.json) from your local computer, and then click OK.

Configure a scheduled workflow

1. On the Workflow Definition page, find the target workflow definition and click Configure Scheduling Time in the Actions column.

2. In the dialog box that appears, set the parameters and click OK.

   • Time Period: The schedule is active only during this time range. No instances are generated outside this range.

   • Configure Scheduling Time: The time and frequency for the workflow to run.

3. In the Schedule Status column, turn on the switch to enable the schedule.

Related topics

• To view information about workflow instances, see Manage workflow instances.

• For more information about supported parameters, see Parameter reference.