To develop and manage Lindorm tasks in DataWorks, associate your Lindorm instance as a computing resource. You can then use the resource for operations such as data synchronization and development.
Background
Lindorm is a distributed computing service built on a cloud-native architecture. It supports community-edition computing models, is compatible with Spark interfaces, and is deeply integrated with the Lindorm storage engine. By leveraging underlying data storage features and indexing capabilities, Lindorm efficiently completes distributed jobs, making it ideal for scenarios such as massive data processing, interactive analysis, machine learning, and graph computing.
Prerequisites
You have created a workspace.
ImportantThis feature is available only in workspaces where Use Data Studio (New Version) is enabled.
You have created a Lindorm instance that meets the following conditions:
The compute engine is enabled for the Lindorm instance.
The Lindorm instance and the DataWorks workspace are in the same region.
You have used a Serverless resource group and bound it to the target DataWorks workspace.
Associate a Lindorm computing resource
Limitations
Supported regions: China (Hangzhou), China (Shanghai), China (Beijing), China (Shenzhen), China (Chengdu), China (Hong Kong), Japan (Tokyo), Singapore, Malaysia (Kuala Lumpur), and Indonesia (Jakarta).
Permission restrictions:
Only DataWorks Serverless resource groups can be used to run Lindorm tasks.
Only members with the O&M or Workspace Administrator role, or the
AliyunDataWorksFullAccesspermission, can create computing resources. For more information about member roles and permissions, see Add workspace members and manage their roles and permissions.
Go to the computing resources page
Go to the DataWorks Workspaces page. In the top navigation bar, switch to the target region and find the workspace where you want to create the computing resource. Click the workspace name or Details in the Operation column to go to the workspace details page.
In the left-side navigation pane, click Computing Resources and select the required computing resource type to go to its list page.
Associate a Lindorm computing resource
On the computing resources page, configure and associate a Lindorm computing resource as described in the following table.
Navigate to the Lindorm association page.
Click Associate Computing Resources to go to the Associate Computing Resources page.
On the Associate Computing Resources page, select Lindorm as the computing resource type. The Associate Lindorm Computing Resource configuration page appears.
Configure the Lindorm computing resource.
On the Associate Lindorm Computing Resource page, configure the parameters as described in the following table.
Section
Parameter
Description
Basic Information
Configuration Mode
Only Alibaba Cloud Instance Mode is supported.
Instance
Select the Lindorm instance to associate with DataWorks. This drop-down list displays all Lindorm instances in the current region.
Database Name
Select the database that DataWorks connects to by default when using this Lindorm computing resource. The default database is
default.Username/Password
Enter the username and password for DataWorks to access this Lindorm computing resource. You can find your credentials in the Lindorm console. Find your Lindorm instance, click the Instance Name, and go to Database Connection in the left-side navigation pane to obtain the username and password.
Computing Resource Instance Name
Enter a custom name for the computing resource instance. At runtime, tasks can use this name to select this computing resource.
Connection Configuration
Connected state
In the connection configuration section, select the Serverless resource group to run Lindorm tasks. Click Test Connectivity to ensure the resource group can access your Lindorm instance. For more information, see Overview of network connectivity solutions.
Click Confirm to complete the configuration of the Lindorm computing resource.
(Optional) Configure global Spark parameters
In DataWorks, you can specify workspace-level Spark parameters for modules such as Data Studio, DataAnalysis, and Operation Center. These modules then use the specified Spark parameters by default to run tasks. For details on customizing global Spark parameters and their precedence over local parameters configured within a specific module, see Set global Spark parameters.
Background
Apache Spark is an engine for large-scale data analysis. In DataWorks, you can configure the Spark parameters used by scheduling nodes at runtime in the following ways:
Method 1: You can configure global Spark parameters to set the workspace-level Spark parameters used by a DataWorks module when it runs Lindorm tasks. You can also define whether this global configuration takes precedence over module-specific configurations. For more information, see Configure global Spark parameters.
Method 2: In Data Studio, you can set specific Spark properties for a node on its editing page. Other product modules do not currently support setting Spark properties internally.
Access control
Only users with the following roles can configure global Spark parameters:
Alibaba Cloud account.
A RAM user or RAM role with the
AliyunDataWorksFullAccesspermission.A RAM user with the Workspace Administrator role.
View global Spark parameters
Go to the computing resources page and find your associated Lindorm computing resource.
Click Spark parameters to open the configuration pane and view the global Spark parameter configuration.
Configure global Spark parameters
You can configure global Spark parameters by following the steps below. To configure Spark parameters for the Lindorm computing resource, see Job configuration.
Go to the computing resources page and find your associated Lindorm computing resource.
Click Spark parameters to open the configuration pane and view the global Spark parameter configuration.
Set global Spark parameters.
In the upper-right corner of the Spark parameters page, click Edit Spark parameters to configure global Spark parameters and their precedence for each module.
NoteThis is a workspace-level global configuration. Before you proceed, make sure you are in the correct workspace.
Parameter
Steps
Spark property
Configure the Spark properties that modules use when running Lindorm tasks. For more information, see Job configuration.
Click Add below, and enter a Spark Property Name and its corresponding Spark Property Value.
NoteTo enable data lineage and output information collection, configure the following:
Set Spark Property Name to
spark.sql.queryExecutionListeners.Set Spark Property Value to
com.aliyun.dataworks.meta.lineage.LineageListener.
For more Spark property settings, see Job configuration.
Global Settings Take Precedence
If you select this option, the global configuration takes precedence over configurations set within product modules. Tasks will consistently use the global Spark properties.
Global configuration: Refers to the Spark properties configured on the Spark parameters page for the Lindorm computing resource under .
Currently, global Spark parameters can be set only for the Data Studio and Operation Center modules.
Module-level configuration:
Data Studio: For Lindorm Spark and Lindorm Spark SQL nodes, you can configure Spark properties for a specific node on the node editing page or in the Configuration Items section.
Other product modules: Do not currently support setting Spark properties within the module.
Click Confirm to save your global Spark parameters.
Next steps
After you configure the Lindorm computing resource, you can use it to develop node tasks in Data Studio. For more information, see Lindorm Spark node and Lindorm Spark SQL node.
When you configure global Spark parameters, enable data lineage and output information collection for Lindorm. After you create and run a metadata collector, you can use Data Map to view and manage Lindorm metadata.