To develop and manage Lindorm tasks in DataWorks, associate your Lindorm instance as a DataWorks Lindorm computing resource. Once associated, the computing resource is available for data synchronization, development, and other operations in DataWorks.
Lindorm is a distributed computing service built on a cloud-native architecture. It supports community-edition computing models, is compatible with Spark interfaces, and deeply integrates with the Lindorm storage engine. It is ideal for scenarios such as large-scale data processing, interactive analysis, machine learning, and graph computing.
Prerequisites
Before you begin, ensure that you have:
A DataWorks workspace created with Use Data Studio (New Version) enabled. See Create a workspace.
A Lindorm instance created with the following conditions met: See Create a Lindorm instance.
The compute engine is enabled for the Lindorm instance.
The Lindorm instance and the DataWorks workspace are in the same region.
A Serverless resource group created and associated with the target DataWorks workspace. See Create a Serverless resource group.
Limitations
Supported regions: China (Hangzhou), China (Shanghai), China (Beijing), China (Shenzhen), China (Chengdu), China (Hong Kong), Japan (Tokyo), Singapore, Malaysia (Kuala Lumpur), and Indonesia (Jakarta).
Resource group: Only DataWorks Serverless resource groups are supported for running Lindorm tasks.
Permissions: Only workspace members with the O&M or Workspace Administrator role, or members with the AliyunDataWorksFullAccess permission, can create computing resources. For details, see Add workspace members and manage their roles and permissions.
Associate a Lindorm computing resource
Go to the computing resources page
Go to the DataWorks Workspaces page. In the top navigation bar, switch to the target region and find the target workspace.
Click the workspace name or click Details in the Operation column to open the workspace details page.
In the left navigation pane, click Computing Resources.
Configure and associate the resource
On the Computing Resources page, click Associate Computing Resource.
On the Associate Computing Resource page, set the computing resource type to Lindorm. The Associate Lindorm Computing Resource page appears.
Configure the following parameters.
Parameter Description Configuration Mode The mode for connecting to the Lindorm instance. Only Alibaba Cloud Instance Mode is supported. Instance The Lindorm instance to associate. The drop-down list shows all Lindorm instances in the current region. Database Name The database that DataWorks connects to by default when using this computing resource. The default database is default.Username / Password The credentials for authentication when DataWorks accesses this Lindorm computing resource. Find the username and password in the Lindorm console: click the instance name, then go to Database Connection in the left navigation pane. Computing Resource Instance Name A custom name for the computing resource. Tasks use this name to identify and select the resource at runtime. Connectivity Status (under Connection Configuration) The Serverless resource group that DataWorks uses to run Lindorm tasks. Click Test Network Connectivity to verify that the resource group can reach the Lindorm instance. For network troubleshooting, see Overview of network connection solutions. Click OK to save the configuration.
(Optional) Configure global Spark-related parameters
In DataWorks, you can set Spark properties at the workspace level. These global settings apply by default across modules — Data Studio, DataAnalysis, and Operation Center — and can take precedence over any module-level Spark configurations. For more information, see Set global Spark-related parameters.
How global and node-level Spark configurations work
Apache Spark is an engine for large-scale data analysis. In DataWorks, you can configure Spark properties that scheduling nodes use at runtime in two ways:
Global (workspace-level): Configure Spark properties on the Spark-related Parameter page for the Lindorm computing resource. These settings apply to all supported modules and can be set to take precedence over module-level configurations. Currently supported for Data Studio and Operation Center.
Node-level (Data Studio only): Set Spark properties for an individual node on the Configuration Items tab of the node editing page. Other modules do not support node-level Spark property configuration.
Access control
Only the following roles can configure global Spark-related parameters:
Alibaba Cloud account
A Resource Access Management (RAM) user or RAM role with the
AliyunDataWorksFullAccesspermissionA RAM user with the Workspace Administrator role
View global Spark-related parameters
Go to the Computing Resources page and find the Lindorm computing resource.
Click Spark-related Parameter to open the configuration pane and view the current global settings.
Set global Spark-related parameters
This is a workspace-level configuration. Make sure you are in the correct workspace before proceeding.
Go to the Computing Resources page and find the Lindorm computing resource.
Click Spark-related Parameter to open the configuration pane.
In the upper-right corner, click Edit Spark-related Parameter and configure the following settings.
Parameter Description Spark Property Spark properties that each module uses when running Lindorm tasks. Click Add, then enter a Spark Property Name and its Spark Property Value. For the full list of supported properties, see Job configuration. To enable data lineage and output collection, add the following property: Spark Property Name = spark.sql.queryExecutionListeners, Spark Property Value =com.aliyun.dataworks.meta.lineage.LineageListener.Global Settings Take Precedence If selected, the global Spark properties override Spark configurations set within individual product modules, and tasks run using only the globally configured properties. The global configuration refers to properties set on the Spark-related Parameter page under Management Center > Computing Resources. Module-level configuration is supported only in Data Studio (on the Configuration Items tab for individual Lindorm Spark or Lindorm Spark SQL nodes); other modules do not support module-level Spark property configuration. Click OK to save.
What's next
Use the Lindorm computing resource to develop nodes in Data Studio. See Lindorm Spark node and Lindorm Spark SQL node.
Enable Lindorm data lineage collection by configuring the global Spark property described above. After you create and run a metadata collector, view and manage Lindorm metadata in Data Map. See View and manage Lindorm in Data Map.