This topic describes how to create an E-MapReduce (EMR) Hive node. This node lets you use SQL-like statements to read, write, and manage data warehouses that store large datasets on a distributed storage system. Use this node to analyze and develop large amounts of log data. - DataWorks

You can create an E-MapReduce (EMR) Hive node. This node lets you use SQL-like statements to read, write, and manage data warehouses that store large datasets on a distributed storage system. Use this node to analyze and develop large amounts of log data.

Prerequisites

An Alibaba Cloud EMR cluster is created and registered to DataWorks. For more information, see DataStudio (old version): Associate an EMR computing resource.
(Required if you use a RAM user to develop tasks) The RAM user is added to the DataWorks workspace as a member and is assigned the Develop or Workspace Administrator role. The Workspace Administrator role has more permissions than necessary. Exercise caution when you assign the Workspace Administrator role. For more information about how to add a member, see Add workspace members and assign roles to them.
A serverless resource group is purchased and configured. The configurations include association with a workspace and network configuration. For more information, see Create and use a serverless resource group.
A workflow is created in DataStudio.
Development operations in different types of compute engines are performed based on workflows in DataStudio. Therefore, before you create a node, you must create a workflow. For more information, see Create a workflow.

Limits

This type of node can be run only on a serverless resource group or an exclusive resource group for scheduling. We recommend that you use a serverless resource group.
If you want to manage metadata for a DataLake or custom cluster in DataWorks, you must first configure EMR-HOOK in your cluster. If EMR-HOOK is not configured, metadata cannot be displayed in real time, audit logs cannot be generated, and data lineages cannot be displayed in DataWorks. In addition, EMR governance tasks cannot be run. For information about how to configure EMR-HOOK, see Configure EMR-HOOK for Hive.

Step 1: Create an EMR Hive node

Go to the DataStudio page.
Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose Data Development and O&M > Data Development. On the page that appears, select the desired workspace from the drop-down list and click Go to Data Development.
Create an EMR Hive node.
1. Right-click the target workflow and choose Create Node > EMR > EMR Hive.
  Note
  Alternatively, you can hover over Create and select Create Node > EMR > EMR Hive.
2. In the Create Node dialog box, configure the Name, Engine Instance, Node Type, and Path parameters and click Confirm. The configuration tab for the EMR Hive node appears.
  Note
  The node name can contain uppercase letters, lowercase letters, Chinese characters, digits, underscores (_), and periods (.).

Step 2: Develop an EMR Hive task

On the configuration page of the EMR Hive node, double-click the node that you created to go to the task development page. Then, perform the development operations.

Develop SQL code

In the SQL editor, you can develop node code. In the code, you can define variables using the ${variable_name} format. You can then assign values to these variables under Scheduling Configuration > Scheduling Parameters in the right-side navigation bar on the node editing page. This lets you dynamically pass parameters to the code in scheduling scenarios. For more information about scheduling parameters, see Supported formats for scheduling parameters. The following is an example.

show tables;
select '${var}'; --You can assign a specific scheduling parameter to the var variable.
select * from userinfo ;

Note

The total size of the SQL statements cannot exceed 130 KB.
If multiple EMR computing resources are attached to the Data Development component in your workspace, you must select a computing resource. If only one EMR computing resource is attached, you do not need to select one.
Click Run With Parameters in the top toolbar to modify parameter assignments in the code. For more information about the parameter assignment logic, see What are the differences in parameter assignment logic among Run, Run with Parameters, and smoke testing in the development environment?.

(Optional) Configure advanced parameters

You can configure node-specific properties in the Advanced Settings section. For more information about property parameters, see Spark Configuration. The available advanced parameters vary based on the EMR cluster type. The following tables describe the parameters.

DataLake or custom cluster: EMR on ECS

Advanced parameter	Description
queue	The scheduling queue to which jobs are committed. Default value: default. For information about EMR YARN, see Basic Queue Configuration.
priority	The priority. The default value is 1.
FLOW_SKIP_SQL_ANALYZE	The method used to execute SQL statements. Valid values: `true`: Executes multiple SQL statements in a single run. `false` (default): Executes one SQL statement at a time. Note This parameter is available only for testing workflows in the development environment.
DATAWORKS_SESSION_DISABLE	This parameter applies to scenarios where you directly run tests in the development environment. Valid values: `true`: A JDBC connection is established every time an SQL statement is executed. `false` (default): The same JDBC connection is reused when different SQL statements are executed within a single node. Note If this parameter is set to `false`, the `yarn applicationId` for Hive is not printed. If you want to print the `yarn applicationId`, set this parameter to `true`.
Others	You can also add custom Hive connection parameters in the advanced configuration.

Hadoop cluster: EMR on ECS

Advanced parameter	Description
queue	The scheduling queue to which jobs are committed. Default value: default. For information about EMR YARN, see YARN schedulers.
priority	The priority. The default value is 1.
FLOW_SKIP_SQL_ANALYZE	The method used to execute SQL statements. Valid values: `true`: Multiple SQL statements are executed in each execution. `false` (default): Only one SQL statement is executed at a time. Note This parameter is available only for testing workflows in the development environment.
USE_GATEWAY	Specifies whether to submit jobs for this node through a gateway cluster. Valid values: `true`: Submits jobs through a Gateway cluster. `false` (default): Use no gateway cluster to commit jobs. Jobs are automatically committed to the master node. Note If the cluster to which this node belongs is not associated with a gateway cluster and you set this parameter to `true`, subsequent EMR job submissions fail.

Run the SQL task

In the toolbar, click the icon. In the Parameters dialog box, select the scheduling resource group that you created and click Run.
Note
- To access a computing resource on the public internet or in a VPC, you need a scheduling resource group that has passed a connectivity test with the computing resource. For more information, see Network connectivity solutions.
- To change the resource group for a task, click the Run With Parameters icon and select a different scheduling resource group.
- When you use an EMR Hive node to query data, a maximum of 10,000 data records can be returned, and the total data size cannot exceed 10 MB.
Click the icon to save the SQL statements.
(Optional) Perform smoke testing.
If you want to perform smoke testing in the development environment, you can do so after you submit the node. For more information, see Perform smoke testing.

Step 3: Configure scheduling properties

If you want the system to periodically run a task on the node, you can click Properties in the right-side navigation pane on the configuration tab of the node to configure task scheduling properties based on your business requirements. For more information, see Overview.

Note

You must configure the Rerun and Parent Nodes parameters on the Properties tab before you commit the task.

Step 4: Deploy the task

After a task on a node is configured, you must commit and deploy the task. After you commit and deploy the task, the system runs the task on a regular basis based on scheduling configurations.

Click the icon in the top toolbar to save the task.
Click the icon in the top toolbar to commit the task.
In the Submit dialog box, configure the Change description parameter. Then, determine whether to review task code after you commit the task based on your business requirements.
Note
- You must configure the Rerun and Parent Nodes parameters on the Properties tab before you commit the task.
- You can use the code review feature to ensure the code quality of tasks and prevent task execution errors caused by invalid task code. If you enable the code review feature, the task code that is committed can be deployed only after the task code passes the code review. For more information, see Code review.

If you use a workspace in standard mode, you must deploy the task in the production environment after you commit the task. To deploy a task on a node, click Deploy in the upper-right corner of the configuration tab of the node. For more information, see Deploy nodes.

What to do next

After you commit and deploy the task, the task is periodically run based on the scheduling configurations. You can click Operation Center in the upper-right corner of the configuration tab of the corresponding node to go to Operation Center and view the scheduling status of the task. For more information, see View and manage auto triggered tasks.

FAQ

Q: Why do I receive a ConnectException error when I run a node?

A: Ensure that the resource group and the cluster have network connectivity. Navigate to the computing resources list page and click Initialize Resource. In the pop-up window, click Re-initialize to complete the initialization.