DataWorks:CDH Spark SQL node

Prerequisites

• An Alibaba Cloud CDH cluster is created and registered to DataWorks. For more information, see Register a CDH or CDP cluster to DataWorks.

• (Required if you use a RAM user to develop tasks) The desired RAM user is added to your DataWorks workspace as a member and is assigned the Develop or Workspace Administrator role. The Workspace Administrator role has more permissions than necessary. Exercise caution when you assign the Workspace Administrator role. For more information about how to add a member, see Add workspace members and assign roles to them.

  Note

  If you use an Alibaba Cloud account, you can skip this operation.

• A Hive data source is added to the workspace, and the data source has passed the network connectivity test. For more information, see Add and manage data sources.

• A CDH Spark SQL node is created.

Procedure

1. On the configuration tab of the CDH Spark SQL node, perform the following operations to develop a task based on the CDH Spark SQL node:

   Develop SQL code

   In the SQL editor, develop task code. You can define variables in the ${Variable name} format in the task code, and configure scheduling parameters in the Scheduling Parameters section of the Properties tab in the right-side navigation pane of the configuration tab of the CDH Spark SQL node to assign the scheduling parameters to the variables as values. This way, the values of the scheduling parameters are dynamically replaced in the node code when the node is scheduled to run. For more information about how to use scheduling parameters. Sample code:

   CREATE TABLE IF NOT EXISTS test_spark.test_lineage_table_f1 (`id` BIGINT, `name` STRING)
   PARTITIONED BY (`ds` STRING);
   CREATE TABLE IF NOT EXISTS test_spark.test_lineage_table_t2 AS SELECT * FROM test_spark.test_lineage_table_f1;
   INSERT INTO test_spark.test_lineage_table_t2 SELECT id,${var} FROM test_spark.test_lineage_table_f1;

   Note

   • In this example, you can create the test_lineage_table_f1 and test_lineage_table_t2 tables in the test_spark database and copy data from the test_lineage_table_f1 table to the test_lineage_table_t2 table. This example is for reference only. You can write code based on your business requirements.

   • The field name is assigned to the ${var} parameter in the code as the value.

   Run the task on the CDH Spark SQL node

   1. On the Debugging Configurations tab in the right-side navigation pane of the configuration tab of the CDH Spark SQL node, configure the Computing Resource and Resource Group parameters.

      1. Select the name of the CDH cluster that you registered in DataWorks for Computing Resource.

      2. Select the resource group for scheduling that passed the network connectivity test between the CDH computing resource. For more information, see Network connectivity solutions.

      Note

      You can also configure the CUs for Computing parameter based on the resources required for task execution. The default value of this parameter is 0.25.

   2. Click Run to run the task on the node.

2. If you want to run the task on the node on a regular basis, configure the scheduling information based on your business requirements. For more information.

3. After the task on the node is configured, commit and deploy the task. For more information.

4. After you deploy the task on the node, view the status of the task in Operation Center. For more information, see Getting started with Operation Center.