DataWorks:EMR Kyuubi node

Prerequisites

• Create an Alibaba Cloud EMR cluster and register it with DataWorks. For more information, see New Data Studio: Attach an EMR compute resource.

• (Optional, for RAM users) The Resource Access Management (RAM) user for task development must be added to the workspace and assigned the Development or Workspace Administrator role (this role includes extensive permissions and must be granted with caution). For more information, see Add workspace members.

  If you are using a root account, skip this step.

Limitations

These tasks can run only on a serverless resource group (recommended) or an exclusive scheduling resource group.

Procedure

1. On the EMR Kyuubi node editing page, perform the following development operations.

   Develop SQL code

   In the SQL editor, develop the task code. You can define variables in your code by using the ${variable_name} format. On the right-side panel, navigate to the Scheduling Settings section and assign values to these variables in the Scheduling Parameters field. This method allows you to pass dynamic parameters to your code during scheduled runs. For more information about scheduling parameters, see Sources and expressions of scheduling parameters. The following code provides an example:

   SHOW TABLES;
   SELECT * FROM kyuubi040702 WHERE age >= '${a}'; -- You can use scheduling parameters.

   Note

   The SQL statement cannot exceed 130 KB in size.

   (Optional) Configure advanced parameters

   On the right-side panel, go to Scheduling Settings > EMR Node Parameters > DataWorks parameters.

   Note

   To configure other open-source Spark properties, go to Scheduling Settings > EMR Node Parameters > Spark parameter on the right-side panel.

   Parameter	Description
   queue	The YARN resource queue for the job. The default queue is default. Note If a workspace-level YARN Resource Queue is configured when you register the EMR cluster with the DataWorks workspace, the queue selection rules for Kyuubi tasks are as follows: If Global Settings Take Precedence is set to Yes, the scheduling queue configured during EMR cluster registration is used. If Global Settings Take Precedence is not configured, the scheduling queue configured for the EMR Kyuubi node is used. For more information about EMR YARN, see Basic queue configurations. For more information about queue configuration during EMR cluster registration, see Set a global YARN resource queue.
   priority	The job priority. The default value is 1.
   FLOW_SKIP_SQL_ANALYZE	Specifies how SQL statements are executed. Valid values: true: Executes multiple SQL statements at a time. false (default): Executes one SQL statement at a time. Note This parameter applies only to test runs in the data development environment.
   DATAWORKS_SESSION_DISABLE	Applies to direct test runs in the development environment. Valid values: true: Creates a new JDBC connection for each SQL statement execution. false (default): Reuses the same JDBC connection when you run different SQL statements within the same node. Note If this parameter is set to false, the Hive yarn applicationId is not printed in the log. To print the yarn applicationId, set this parameter to true.

   Run the SQL task

   1. In the Compute Resource section of Run Configuration, select the Compute Resource and DataWorks Resource Group.

      Note

      • You can also adjust the CUs for Scheduling based on the resource requirements of the task. The default value is 0.25.

      • To access a data source over the public network or in a VPC, you must use a scheduling resource group with verified connectivity to that data source. For more information, see Network connectivity solutions.

   2. In the parameter dialog box in the toolbar, select the data source, and then click Run.

2. To run node tasks on a schedule, configure the scheduling information based on your business requirements. For more information, see Node scheduling configuration.

3. After you configure the node, deploy it. For more information, see Deploy nodes and workflows.

4. After a task is deployed, you can view its run status in Operation Center. For more information, see Get started with Operation Center.

FAQ

• Q: The node fails with a connection timeout error. What can I do?

  A: Ensure that the resource group and the cluster can connect to each other over the network. Go to the computing resources list and click Resource Initialization. In the dialog box that appears, click Re-initialize. Verify that the initialization is successful.

  A message at the top of the dialog box indicates that you must initialize the resource group when you bind the cluster for the first time or when cluster service configurations change, such as modifications to hive-site.xml. Otherwise, tasks may fail. After the initialization is successful, the status column shows Successfully initialized and displays the completion time.