Create an EMR Shell node

You can create E-MapReduce (EMR) Shell nodes in DataWorks to meet specific business requirements. You can specify custom Shell script and run the script to use features such as data processing, Hadoop component calling, and file management. This topic describes how to configure and use an EMR Shell node in DataWorks to specify and run a Shell script.

Background information

You can use the OSS REF method to reference OSS resources. For more information, see the Method 1: Reference an OSS resource section in this topic.

Prerequisites

A workflow is created in DataStudio.
Development operations in different types of compute engines are performed based on workflows in DataStudio. Therefore, before you create a node, you must create a workflow. For more information, see Create a workflow.
An Alibaba Cloud EMR cluster is created and registered to DataWorks.
Before you create an EMR node and use the EMR node to develop EMR tasks in DataWorks, you must register a desired EMR cluster to a DataWorks workspace. For more information, see Register an EMR cluster to DataWorks.
(Required if you use a RAM user to develop tasks) The RAM user is added to the DataWorks workspace as a member and is assigned the Development or Workspace Administrator role. The Workspace Administrator role has more permissions than necessary. Exercise caution when you assign the Workspace Administrator role. For more information about how to add a member, see Add workspace members and assign roles to them.
An exclusive resource group for scheduling is purchased and configured. The configurations include association with a workspace and network configuration. For more information, see Create and use an exclusive resource group for scheduling.

Limits

Tasks on this type of node cannot be run on the shared resource group for scheduling. You can run the tasks only on exclusive resource groups that are created after December 1, 2023.
If you want to manage metadata for a DataLake or custom cluster in DataWorks, you must configure EMR-HOOK in the cluster first. If you do not configure EMR-HOOK in the cluster, metadata cannot be displayed in real time, audit logs cannot be generated, and data lineages cannot be displayed in DataWorks. In addition, the related EMR governance tasks cannot be run. For more information about how to configure EMR-HOOK, see Use the Hive extension feature to record data lineage and historical access information.

Precautions

If you commit a node by using spark-submit, we recommend that you set deploy-mode to cluster rather than client.
EMR Shell nodes are run on the resource groups for scheduling of DataWorks rather than in EMR clusters. You can run specific commands supported by EMR components but cannot directly read the information about EMR resources. If you want to reference an EMR resource in an EMR Shell node, you must upload the resource to DataWorks first. For more information, see Create and use an EMR JAR resource.

Go to the DataStudio page.
Log on to the DataWorks console. In the left-side navigation pane, choose Data Modeling and Development > DataStudio. On the page that appears, select the desired workspace from the drop-down list and click Go to DataStudio.
Go to the DataStudio page.
Log on to the DataWorks console. In the left-side navigation pane, choose Data Modeling and Development > DataStudio. On the page that appears, select the desired workspace from the drop-down list and click Go to DataStudio.
Create an EMR Shell node.
1. Find the desired workflow, right-click the name of the workflow, and then choose Create Node > EMR > EMR Shell.
  Note
  Alternatively, you can move the pointer over the icon and choose Create Node > EMR > EMR Shell.
2. In the Create Node dialog box, configure the Name, Engine Instance, Node Type, and Path parameters.
  Note
  The node name must be 1 to 128 characters in length and can contain only letters, digits, underscores (_), and periods (.).
3. Click Confirm. The configuration tab of the EMR Shell node appears.

Develop a Shell task

Use one of the following methods to develop a Shell task on the configuration tab of the EMR Shell node:

Method 1: Reference an OSS resource

The current node can reference an OSS resource by using the OSS REF method. When you run a task on the node, DataWorks automatically loads the OSS resource specified in the node code. This method is commonly used in scenarios in which JAR dependencies are required in EMR tasks or EMR tasks need to depend on scripts. Reference format:

ossref://{endpoint}/{bucket}/{object}

endpoint: the endpoint of OSS. If the endpoint parameter is left empty, only a resource in an OSS bucket that resides in the same region as the current EMR cluster can be referenced.
bucket: a container that is used to store objects in OSS. Each bucket has a unique name. You can log on to the OSS console to view all buckets within the current logon account.
object: a file name or path that is stored in a bucket.

Note

This feature is unavailable for the shared resource group for scheduling. To use this feature on a resource group that is created before December 1, 2023, you must contact technical support to upgrade the configurations of the resource group.

Method 2: Upload and reference an EMR JAR resource

DataWorks allows you to upload a resource from your on-premises machine to DataStudio before you can reference the resource. If the EMR cluster that you want to use is a DataLake cluster, you can perform the following steps to reference an EMR JAR resource. If the EMR Shell node depends on large amounts of resources, the resources cannot be uploaded by using the DataWorks console. In this case, you can store the resources in Hadoop Distributed File System (HDFS) and reference the resources in the code of the EMR Shell node.

Create an EMR JAR resource.
For more information about how to create an EMR JAR resource, see Create and use an EMR JAR resource. In this example, the JAR package that is generated in the Prepare initial data and a JAR resource package section is stored in the emr/jars directory. The directory is used to store JAR resources. The first time you use an EMR JAR resource, click Authorize to authorize DataWorks to access the EMR JAR resource. Then, click Upload to upload the JAR resource.
Reference the EMR JAR resource.
1. Open the EMR Shell node. The configuration tab of the node appears.
2. Find the resource that you want to reference below Resource in the EMR folder, right-click the resource name, and then select Insert Resource Path. In this example, the resource is onaliyun_mr_wordcount-1.0-SNAPSHOT.jar.
3. If the information in the ##@resource_reference{""} format appears on the configuration tab of the EMR Shell node, the code resource is referenced. Then, run the following code. You must replace the information in the following code with the actual information. The information includes the resource package name, bucket name, and directory.
```
##@resource_reference{"onaliyun_mr_wordcount-1.0-SNAPSHOT.jar"}
onaliyun_mr_wordcount-1.0-SNAPSHOT.jar cn.apache.hadoop.onaliyun.examples.EmrWordCount oss://onaliyun-bucket-2/emr/datas/wordcount02/inputs oss://onaliyun-bucket-2/emr/datas/wordcount02/outputs
```
  Note
  You cannot add comments when you write code for the EMR Shell node.

Configure scheduling parameters for the EMR Shell node

In the SQL editor, enter the code for the EMR Shell node. Example:

DD=`date`;
echo "hello world, $DD"
## Scheduling parameters are supported.
echo ${var};

If you use an EMR DataLake cluster, the following command lines are also supported:

Shell commands: Shell commands under /usr/bin and /bin, such as ls and echo.
YARN: hadoop, hdfs, and yarn.
Spark: spark-submit.
Sqoop: sqoop-export, sqoop-import, and sqoop-import-all-tables.
Note
To use the Sqoop service, you must add the IP address of your resource group to the IP address whitelist of the ApsaraDB RDS instance that is used to store the metadata of the EMR cluster.

If you want to change the scheduling parameter that is assigned to the variable in the code, click Run with Parameters in the top toolbar. For information about value assignment for the scheduling parameters, see What are the differences in the value assignment logic of scheduling parameters among the Run, Run with Parameters, and Perform Smoke Testing in Development Environment modes?

Note

For more information about how to use scheduling parameters, see Supported formats of scheduling parameters.

Configure scheduling properties

If you want the system to periodically run the node, you can click Properties in the right-side navigation pane on the configuration tab of the node to configure properties for the node based on your business requirements.

Configure basic properties for the node. For more information, see Configure basic properties.
Configure the scheduling cycle, rerun properties, and scheduling dependencies of the node. For more information, see Configure time properties and Configure same-cycle scheduling dependencies.
Note
Before you commit the node, you must configure the Rerun and Parent Nodes parameters on the Properties tab.
Configure resource properties for the node. For more information, see Configure the resource property. If the node that you created is an auto triggered node and you want the node to access the Internet or a virtual private cloud (VPC), you must select the resource group for scheduling that is connected to the node. For more information, see Establish a network connection between a resource group and a data source.

Debug the node

Optional. Select a resource group and assign custom parameters to variables.
- Click the icon in the top toolbar of the configuration tab of the EMR Presto node. In the Parameters dialog box, select an exclusive resource group for scheduling that you want to use to debug and run the node.
- If you use scheduling parameters in your node code, you can assign the scheduling parameters to variables as values in the node code for debugging. For more information about the assignment logic of scheduling parameters, see What are the differences in the value assignment logic of scheduling parameters among the Run, Run with Parameters, and Perform Smoke Testing in Development Environment modes?
Save and execute the SQL statements.
In the top toolbar, click the icon to save the SQL statements. Then, click the icon to execute the SQL statements.
Optional. Perform smoke testing.
You can perform smoke testing on the node in the development environment when you commit the node or after you commit the node. For more information, see Perform smoke testing.