Use the LHM scheduling migration tool to import a custom DataWorks Spec into DataWorks.
1. Construct and parse the DataWorks Spec
The LHM migration tool allows you to create description files based on the standard DataWorks workflow structure (DataWorks Spec) and import them. Before the import, LHM parses the DataWorks Spec and generates a standard migration package.
After the standard migration package is generated, no further transformation is required. Run the import tool directly to write the workflows to DataWorks.
1.1 DataWorks Spec definition
https://github.com/aliyun/dataworks-spec/tree/master
1.2 Build the input package
-
Create workflow definitions based on the DataWorks Spec definition and save them as JSON files.
-
Place all workflow JSON files in a single folder without subfolders. Compress the folder into a .zip file.
{
"version": "1.1.0",
"kind": "CycleWorkflow",
"spec": {
"nodes": [],
"workflows": [
{
"id": "3451387436863448",
"outputs": {
"nodeOutputs": [
{
"artifactType": "NodeOutput",
"data": "3451387436863448",
"refTableName": "example_hive_operator"
}
]
},
"nodes": [
{
"id": "2105761738722077",
"name": "run_first",
"instanceMode": "T+1",
"rerunMode": "FailureAllowed",
"rerunTimes": 0,
"rerunInterval": 300000,
"trigger": {
"type": "Scheduler",
"startTime": "2025-03-30 00:00:00",
"timezone": "UTC"
},
"runtimeResource": {
"resourceGroup": "Serverless_res_group_651147510078336_710919704558240"
},
"script": {
"path": "run_first",
"runtime": {
"command": "EMR_HIVE"
},
"parameters": [],
"content": "\n create database if not exists airflow;\n use airflow;\n drop table if exists test_hive;\n create table test_hive(name string);\n insert into test_hive values('studio');\n "
},
"outputs": {
"nodeOutputs": [
{
"data": "2105761738722077"
},
{
"artifactType": "NodeOutput",
"data": "example_hive_operator.run_first.run_second",
"refTableName": "run_first"
}
]
},
"type": "EMR_HIVE"
},
{
"id": "3326558787158921",
"name": "run_second",
"instanceMode": "T+1",
"rerunMode": "FailureAllowed",
"rerunTimes": 0,
"rerunInterval": 300000,
"trigger": {
"type": "Scheduler",
"startTime": "2025-03-30 00:00:00",
"timezone": "UTC"
},
"runtimeResource": {
"resourceGroup": "Serverless_res_group_651147510078336_710919704558240"
},
"script": {
"path": "run_second",
"runtime": {
"command": "EMR_HIVE"
},
"parameters": [],
"content": "\n use airflow;\n add jar oss://emr-studio-example/hive-udf-1.0-SNAPSHOT.jar;\n create temporary function simpleudf AS 'com.aliyun.emr.hive.udf.SimpleUDFExample';\n show functions like '*udf';\n select simpleudf(name) from test_hive;\n "
},
"outputs": {
"nodeOutputs": [
{
"data": "3326558787158921"
}
]
},
"type": "EMR_HIVE"
}
],
"dependencies": [
{
"nodeId": "3326558787158921",
"depends": [
{
"type": "Normal",
"output": "2105761738722077",
"refTableName": "run_first"
}
]
}
],
"script": {
"path": "Airflow_Import_V3/example_hive_operator",
"runtime": {
"command": "WORKFLOW"
},
"parameters": []
},
"name": "example_hive_operator",
"trigger": {
"type": "Scheduler",
"cron": "0 0 0 * * ?",
"timezone": "UTC",
"delaySeconds": 0
},
"type": "CycleWorkflow",
"strategy": {
"timeout": 0,
"instanceMode": "T+1",
"rerunMode": "Allowed",
"rerunTimes": 0,
"rerunInterval": 0,
"failureStrategy": "Continue"
}
}
],
"flow": [
{
"nodeId": "3451387436863448",
"depends": []
}
]
}
}Example compression command
zip -q -r -m -o <PackageName>.zip PackageName1.3 Build the configuration items
Use the following configuration items as-is. Because scheduling information is already described in the custom DataWorks Spec, few additional configuration items are needed.
{
"schedule_datasource": {
"name": "MySpec",
"type": "DataWorks",
"operaterType": "MANUAL"
},
"conf": {}
}1.4 Run the scheduling parsing tool
Run the parsing tool from the command line using the following command:
sh ./bin/run.sh read \
-c ./conf/<your_config_file>.json \
-f ./data/0_OriginalPackage/<input_package>.zip \
-o ./data/1_ReaderOutput/<standard_migration_package>.zip \
-t dw-newidea-readerThe -c parameter specifies the configuration file path. The -f parameter specifies the input package path. The -o parameter specifies the output path for the standard migration package. The -t parameter specifies the discovery plugin name.
For example, to parse project A:
sh ./bin/run.sh read \
-c ./conf/projectA_read.json \
-f ./data/0_OriginalPackage/projectA_DataworksSpec.zip \
-o ./data/1_ReaderOutput/projectA_ReaderOutput.zip \
-t dw-newidea-readerThe discovery tool displays process information during execution. Check the output for any errors.
1.5 View the parsing results
Open the generated ReaderOutput.zip package in the ./data/1_ReaderOutput/ directory to preview the parsing results.
The statistical report summarizes workflows, nodes, resources, functions, and data sources from the DataWorks Spec.
The data/project folder contains the standardized data structure of the scheduling information from the DataWorks Spec.
The statistical report provides two special features:
1. You can change some workflow and node properties in the report. Editable fields are shown in blue. The tool applies these property changes when importing data into DataWorks.
2. You can skip workflows during import by deleting their corresponding rows from the workflow child table in the report. This acts as a workflow blacklist. Note: Workflows with dependencies on each other must be imported in the same batch. Do not separate them using the blacklist, because this can cause errors.
For more information, see Using the scheduling migration overview report to update scheduling properties.
2. Import into DataWorks
The import tool supports multiple write operations and automatically creates or updates workflows in overwrite mode.
2.1 Prerequisites
2.1.1 The DataWorks Spec is parsed successfully
The DataWorks Spec is parsed and the ReaderOutput.zip file is generated.
(Optional but recommended) Open the parsing output package and review the statistical report to verify that the migration scope was parsed correctly.
2.1.2 Configure DataWorks
Perform the following actions in DataWorks:
1. Create a workspace.
2. Create an AccessKey pair (AK and SK) and grant the pair administrator permissions for the workspace. We strongly recommend creating an AccessKey pair that is bound to your account to simplify troubleshooting for write operations.
3. In the workspace, create data sources, attach computing resources, and create resource groups.
4. In the workspace, upload file resources and create User-Defined Functions (UDFs).
2.1.3 Check network connectivity
Verify the connection to the DataWorks Endpoint.
List of endpoints:
ping dataworks.aliyuncs.com2.2 Configure the import
In the `conf` folder of your project directory, create a configuration file in JSON format, such as `writer.json`.
-
Before you use the file, delete the comments from the JSON code.
{
"schedule_datasource": {
"name": "YourDataWorks", // Give your DataWorks data source a name.
"type": "DataWorks",
"properties": {
"endpoint": "dataworks.cn-hangzhou.aliyuncs.com", // Service endpoint
"project_id": "YourProjectId", // Workspace ID
"project_name": "YourProject", // Workspace name
"ak": "************", // AK
"sk": "************" // SK
},
"operaterType": "MANUAL"
},
"conf": {
"di.resource.group.identifier": "Serverless_res_group_***_***", // Scheduling resource group
"resource.group.identifier": "Serverless_res_group_***_***", // Data Integration resource group
"dataworks.node.type.xls": "/Software/bwm-client/conf/CodeProgramType.xls", // Path to the DataWorks node type table
"qps.limit": 5 // QPS limit for API requests to DataWorks
}
}2.2.1 Endpoint
Select the endpoint based on the region of your DataWorks workspace. For more information, see the following document:
2.2.2 Workspace ID and name
Open the DataWorks console and go to the workspace details page. You can find the workspace ID and name in the Basic Information section on the right.
2.2.3 Create and authorize an AccessKey pair
Create an AccessKey pair (AK and SK) on your user page. The AccessKey pair must have administrator read and write permissions for the target DataWorks workspace.
Permissions are managed in two locations. If you use a RAM user, you must first grant the RAM user permissions to perform DataWorks operations.
Access Policy page: https://ram.console.alibabacloud.com/policies
Then, in the DataWorks workspace, assign workspace permissions to the account.
Note: You can set a network access policy for your AccessKey. Ensure that the IP address of the machine running the migration tool is included in the access policy.
2.2.4 Resource groups
On the DataWorks workspace details page, go to the Resource Groups page from the menu bar on the left. Attach a resource group and obtain its ID.
A general-purpose resource group can be used for both node scheduling and Data Integration. You can set both the scheduling resource group (`resource.group.identifier`) and the Data Integration resource group (`di.resource.group.identifier`) to the same general-purpose resource group.
2.2.5 QPS settings
The tool imports data through the DataWorks API. Different DataWorks editions have different QPS limits and daily call quotas for read and write OpenAPI operations. For more information, see Limitations.
For DataWorks Basic, Standard, and Professional Editions, you can set `"qps.limit"` to `5`. For the Enterprise Edition, you can set `"qps.limit"` to `20`.
Note: Do not run multiple import tools at the same time.
2.2.6 DataWorks node type ID settings
In DataWorks, some node types have different TypeIds across regions. The specific TypeID is determined by the DataWorks Data Development interface. This applies mainly to database nodes. For more information, see Database nodes.
For example, the MySQL node has a NodeTypeId of 1000039 in the China (Hangzhou) region and 1000041 in the China (Shenzhen) region.
To handle these regional differences, the tool allows you to configure the node TypeId table it uses.
Import the table by adding a configuration item in the import tool:
"conf": {
"dataworks.node.type.xls": "/Software/bwm-client/conf/CodeProgramType.xls" // Path to the DataWorks node type table
}To obtain a node type ID from the DataWorks Data Development interface, create a workflow and add a node. After saving the node, view the workflow's Spec.
If the node type is configured incorrectly, the following error appears when you publish the workflow.
2.3 Run the DataWorks import tool
Run the import tool from the command line using the following command:
sh ./bin/run.sh write \
-c ./conf/<your_config_file>.json \
-f ./data/1_ReaderOutput/<parsing_result_output_package>.zip \
-o ./data/4_WriterOutput/<import_result_storage_package>.zip \
-t dw-newide-writerThe -c parameter specifies the configuration file path. The -f parameter specifies the ReaderOutput package path. The -o parameter specifies the output path for the WriterOutput package. The -t parameter specifies the submission plugin name.
For example, to import project A into DataWorks:
sh ./bin/run.sh write \
-c ./conf/projectA_write.json \
-f ./data/1_ReaderOutput/projectA_ReaderOutput.zip \
-o ./data/4_WriterOutput/projectA_WriterOutput.zip \
-t dw-newide-writerThe import tool displays process information during execution. Check the output for any errors. After the import, the command line shows statistics for successful and failed imports. If some nodes fail to import, the overall process is not affected. You can manually modify failed nodes in DataWorks.
2.4 View the import results
After the import, view the results in DataWorks. You can monitor the import process as each workflow is imported. To stop the import, run the `jps` command to find `BwmClientApp`, and then use `kill -9` to stop the process.
5 Q&A
5.1 Development is ongoing at the source. How can I submit incremental changes to DataWorks?
The migration tool runs in overwrite mode. To submit incremental changes from the source to DataWorks, rerun the export, transformation, and import processes. The tool matches workflows by full path to determine whether to create or update a workflow. To migrate changes correctly, do not move the workflows.
5.2 Development is ongoing at the source, and I am also modifying and managing workflows in DataWorks. Will an incremental migration overwrite my changes in DataWorks?
Yes. The migration tool runs in overwrite mode. Make changes in DataWorks only after the entire migration is complete. Alternatively, migrate in batches. After a batch of workflows is migrated and confirmed not to be overwritten again, you can begin modifying those workflows in DataWorks. Different batches do not affect each other.
5.3 The full package import takes too long. Can I import only a part of it?
Yes. Perform a partial import by manually editing the import package. In the `data/project/workflow` folder, keep only the workflows you want to import and delete the rest. Then re-compress the folder and run the import tool. Note that workflows with dependencies on each other must be imported together. Otherwise, the node lineage between workflows is lost.