E-MapReduce:Manage workflow definitions

Prerequisites

Before you begin, ensure that you have:

• A project created in EMR Studio. For more information, see Create a project.

Create a workflow

1. Log on to the E-MapReduce (EMR) console.

2. In the left-side navigation pane, choose EMR Studio > Workflow.

3. Click the Project tab, then click the name of a project.

4. In the left-side navigation pane of the project details page, choose Workflow > Workflow Definition.

5. On the Workflow Definition page, click Create Workflow.

6. On the Create Workflow page, drag a node type onto the canvas. In this example, drag HIVECLI to the canvas.

   In the Current node settings dialog box, configure the parameters and click Confirm. For more information about HIVECLI, see HIVECLI. For other node types, see Node types.

7. (Optional) Configure dependencies between nodes.

   EMR Workflow allows you to configure custom node dependencies between workflows.

   • To connect two nodes: hover over the icon on the right side of a node, then drag the connection line to another node.

   • To delete a dependency or node: click the connection line or node, then click the icon in the upper-right corner of the canvas.

8. Save the workflow.

   1. Click Save in the upper-right corner of the canvas.

   2. In the Basic Information dialog box, configure the following parameters and click Confirm.

      Parameter	Description
      Workflow Name	The name of the workflow.
      Description	A description of the workflow.
      Timeout Alert	Disabled by default. If enabled, specify a timeout period. An alert is triggered when a node's execution time exceeds the timeout period.
      Process execute type	How multiple instances of the same workflow are run concurrently. parallel: instances run at the same time. Serial wait: instances run one after another.
      Global Variables	Variables that apply to all nodes in the workflow.

Workflow states and available operations

A workflow is in either the Online or Offline state. The state determines which operations are available. The following table maps each operation to the state in which it can be performed.

Run a workflow

Each run generates a workflow instance, which appears on the Workflow Instance page.

1. On the Workflow Definition page, find the workflow and click the (Online) icon in the Operation column to bring the workflow online.

2. Click the (Start) icon in the Operation column.

3. In the dialog box that appears, configure the following parameters and click Confirm.

   Parameter	Description
   Failure Strategy	What happens to concurrent nodes when one node fails. Continue: other nodes keep running. End: downstream nodes of the failed node are terminated.
   Notification Strategy	When to send a notification after the workflow ends. Valid values: None, Success, Failure, All.
   Workflow Priority	The priority of the workflow run. Default: MEDIUM. Valid values: HIGHEST, HIGH, MEDIUM, LOW, LOWEST.
   Execution Cluster	The cluster to run the workflow on. Select a cluster associated on the Cluster Manage page of the Security tab.
   Alarm Group	The alarm group for notifications. Select a group configured on the Alarm Group Manage page of the Security tab.
   Complement Data	Whether to generate backfill data for a past time range. See the Complement Data parameters section below.
   Startup Parameter	A startup parameter and value. Defines or overwrites a global variable when this workflow instance starts.
   Whether Dry-Run	When enabled, the workflow performs a dry run and records a success log without executing actual tasks.

4. In the left-side navigation pane of the project details page, choose Workflow > Workflow Instance to view the status of the workflow instance.

Import a workflow

Import a workflow from a JSON file previously exported from EMR Workflow.

1. On the Workflow Definition page, click Import Workflow.

2. In the Upload dialog box, click Upload and select the exported JSON file.

3. Click Confirm.

Schedule a workflow

Configure a Cron-based schedule to run the workflow automatically at a set interval.

1. On the Workflow Definition page, find the workflow and click the (Timing) icon in the Operation column.

   The Timing operation is available only for Online workflows.

2. Configure the following parameters and click Confirm.

   Parameter	Description
   Start and stop time	The time range during which the workflow is scheduled to run. No scheduled instances are generated outside this range.
   Timing	The scheduling interval (Cron expression) at which the workflow runs.
   Execution Cluster	The cluster to use for scheduled runs.

3. Activate the scheduled workflow.

   After saving the scheduling settings, the scheduled workflow entry is in the Offline state. To make it take effect:

   1. On the Workflow Definition page, click the (Cron manage) icon in the Operation column.

   2. On the Cron manage page, find the scheduled workflow entry and click the (Online) icon in the Operation column.

   The scheduled workflow is now active and will run according to the configured interval.

What's next

• Node types

• HIVECLI

• Create a project