DataWorks:CDH Hive node

Prerequisites

• You have created an Alibaba Cloud CDH cluster and associated it with a DataWorks workspace. For more information, see Associate a CDH compute engine.

  Important

  You have installed the Hive component on your CDH cluster and configured the Hive connection information when you associated the cluster.

• (Optional, for RAM users) The RAM user for task development has been added to the corresponding workspace and granted the Development or Workspace Administrator role. The Workspace Administrator role has extensive permissions, so grant it with caution. For more information about adding members, see Add members to a workspace.

  Note

  If you use an Alibaba Cloud account, you can skip this step.

• You have configured a Hive data source in DataWorks and verified that the data source passed the connectivity test. For more information, see Manage data sources.

Limitation

This type of task can be run on serverless resource groups (recommended) or legacy exclusive resource groups for scheduling.

Create a node

For instructions, see Create a node.

Develop the node

Write your task code in the SQL editor. You can define variables in the code using the ${variable_name} format. Then, on the right side of the node editing page, assign a value to the variable in the Scheduling Parameters section of the Scheduling Settings pane. This lets you dynamically pass parameters to your code during scheduling. For more information about scheduling parameters, see Sources and expressions of scheduling parameters. The following code provides an example.

SHOW TABLES;

SELECT * FROM userinfo ;
-- This can be used with scheduling parameters.
SELECT '${var}'; 

Debug the node

1. On the Run Configuration tab, in the Compute Resource section, configure the Compute Resource and Resource Group.

   1. For Compute Resource, select the CDH cluster you registered in DataWorks.

   2. For Resource Group, select a scheduling resource group that has passed the connectivity test with the data source. For more information, see Network connectivity solutions.

2. On the toolbar at the top of the node editing page, click Run to run the task.

Next steps

• Node scheduling configuration: If you need to run a node periodically, configure its Scheduling Policy in the Scheduling Settings panel on the right.

• Publish a node: To run a task in the production environment, click the icon to publish the node. A node runs on schedule only after it is published to the production environment.

• Task O&M: After a task is published, you can monitor the status of its periodic runs in the Operation Center. For more information, see Get started with Operation Center.