DataWorks:Lindorm Spark SQL node

Background information

Lindorm is a distributed computing service based on the cloud-native architecture. It supports Community Edition computing models, is compatible with Spark interfaces, and deeply integrates Lindorm storage engine characteristics. You can use Lindorm to complete distributed jobs and tasks in an efficient manner by leveraging the data storage characteristics and indexing capabilities at the underlying layer. Lindorm can be used in scenarios such as massive data processing, interactive analysis, machine learning, and graph computing.

Create a Lindorm Spark SQL node

For information about how to create a Lindorm Spark SQL node, see Create a Lindorm Spark SQL node.

Configure the Lindorm Spark SQL node

When you write code for the Lindorm Spark SQL node in the code editor, you can define variables in the ${variable name} format. Then, you can assign values to the variables on the Debugging Configurations or Properties tab. Sample code for creating a Lindorm table and inserting data into the table:

CREATE TABLE IF NOT EXISTS lindorm_table_job (
  id INT,
  name STRING,
  data STRING
)
USING parquet
PARTITIONED BY (partition_date DATE);

INSERT OVERWRITE TABLE lindorm_table_job PARTITION (partition_date='${var}')
VALUES (1, 'Alice', 'Sample data 1'), (2, 'Bob', 'Sample data 2');

For more information about Lindorm Spark SQL operations, see SQL reference.

Debug the Lindorm Spark SQL node

1. Configure properties for debugging the Lindorm Spark SQL node.

   In the right-side navigation pane of the configuration tab of the Lindorm Spark SQL node, click Debugging Configurations. On the Debugging Configurations tab, configure the parameters described in the following table.

   Parameter	Description
   Computing Resource	Select the Lindorm computing resource that you associate with the workspace.
   Lindorm Resource Group	Select the Lindorm resource group that you specify when you associate the Lindorm computing resource with the workspace.
   Resource Group	Select the resource group that has passed the connectivity test when you associate the Lindorm computing resource with the workspace.
   Script Parameters	If you define variables in the ${Parameter name} format when you configure the Lindorm Spark SQL node, you must configure the Parameter Name and Parameter Value parameters in the Script Parameters section. When the Lindorm Spark SQL node is run, the parameters that you configure will be dynamically replaced with actual values. For more information, see Supported formats of scheduling parameters.
   Spark Parameters	The Spark configuration parameters. For more information about parameter settings, see Configure parameters for jobs.

2. Debug and run the Lindorm Spark SQL node.

   In the top toolbar of the configuration tab of the Lindorm Spark SQL node, click Save and Run to save and run the node.

What to do next

• Node scheduling: If you want the system to periodically schedule a node in a workspace directory, you can click Properties in the right-side navigation pane of the configuration tab of the node and configure scheduling properties for the node in the Scheduling Policies section.

• Node deployment: If you want to deploy a node to the production environment for running, you can click the icon in the top toolbar of the configuration tab of the node to initiate a deployment process. Nodes in a workspace directory can be periodically scheduled only if they are deployed to the production environment.