Migrate DolphinScheduler to DataWorks | LHM Migration Tool - Migration Hub

Use the LHM migration tool to migrate DolphinScheduler workflows to DataWorks in three steps: export, convert, and import.

1. Export DolphinScheduler workflows

The export tool calls DolphinScheduler APIs to retrieve project, workflow, data source, and resource file information. It supports versions 1.x, 2.x, and 3.x.

1. Prerequisites

Prepare a JDK 17 environment with network access to DolphinScheduler. Download and extract the migration tool.

To test connectivity, call the DolphinScheduler ListProject API and verify that the response contains your target project. Token details are covered in the next section.

# DolphinScheduler 1.x
curl -H "token:<YourToken>" -X GET http://<YourIp>:12345/dolphinscheduler/projects/query-project-list

# DolphinScheduler 2.x
curl -H "token:<YourToken>" -X GET http://<YourIp>:12345/dolphinscheduler/projects/list

# DolphinScheduler 3.x
curl -H "token:<YourToken>" -X GET http://<YourIp>:12345/dolphinscheduler/projects/list

2. Configure connection information

In the `conf` folder, create a JSON configuration file (e.g., `read.json`).

Delete the comments from the JSON file before use.

{
  "schedule_datasource": {
    "name": "YourDolphin", // Give your DolphinScheduler data source a name.
    "type": "DolphinScheduler", // Data source type (DolphinScheduler)
    "properties": {
      "endpoint": "http://localhost:12345", // Endpoint
      "project": "Comprehensive Test", // Project name
      "token": "***********************" // Token
    },
    "operaterType": "AUTO" // Connection type (AUTO: Automatically get scheduling information through the API)
  },
  "conf": {

  }
}

2.1. Get the endpoint

The endpoint is typically the frontend page address, such as `http://120.55.X.XXX:12345`.

If the DolphinScheduler address is `http://your-company:12345/dolphinscheduler/ui/home`, the endpoint is `http://your-company:12345`.

The API module may be customized. If a call fails, check the Swagger page to verify API attributes.

2.2. Get the token

On the Token Manage page under Security, create a token with a long expiration time.

The token must have permissions for the target migration project.

2.3. Get the project

On the Project page, copy the target project name and enter it as the `Project Name` value.

3. Run the scheduling discovery tool

Each run of the discovery tool generates two files:

Raw DolphinScheduler API output (ApiOutput package).
Standardized data structure parsed by the discovery tool (ReaderOutput package).

ReaderOutput is the final export. ApiOutput is for troubleshooting only.

Run the discovery tool:

sh ./bin/run.sh read \
-c ./conf/<your_config_file>.json \
-f ./data/0_OriginalPackage/<api_raw_info_package>.zip \
-o ./data/1_ReaderOutput/<source_discovery_export_package>.zip \
-t <PluginName>

Parameters: `-c` config file path, `-f` ApiOutput storage path, `-o` ReaderOutput storage path, `-t` discovery plugin name.

Export plugins: `dolphinv1-reader` (1.x), `dolphinv2-reader` (2.x), `dolphinv3-reader` (3.x).

For example, to export Project A from DolphinScheduler 3.2.0:

sh ./bin/run.sh read \
-c ./conf/projectA_read.json \
-f ./data/0_OriginalPackage/projectA_ApiOutput.zip \
-o ./data/1_ReaderOutput/projectA_ReaderOutput.zip \
-t dolphinv3-reader

4. View the export results

Open `ReaderOutput.zip` in `./data/1_ReaderOutput/` to preview results.

The statistical report summarizes workflows, nodes, resources, functions, and data sources.

The `data/project` folder contains the standardized scheduling data.

Statistical report:

Sheet 1 ('Overview') summarizes the export results. Subsequent sheets ('WORKFLOW', 'WORKFLOWNODE', etc.) list details on workflows, nodes, resources, functions, and data sources.

The statistical report provides two features:

1. You can edit certain workflow and node properties (shown in blue font). The conversion tool applies these changes during initialization.

2. Delete rows in the workflow sub-table to skip workflows during conversion (workflow blacklist). Interdependent workflows must be converted together — separating them causes errors.

Use the overview report in scheduling migration to add or modify scheduling properties.

5. Q&A

5.1. (Batch discovery) Can I discover multiple projects at once?

Yes. Enter comma-separated project names in the `project` field (no spaces between names; spaces within names are preserved).

Delete the comments from the JSON file before use.

{
  "schedule_datasource": {
    "name": "YourDolphin", // Give your DolphinScheduler data source a name.
    "type": "DolphinScheduler", // Data source type (DolphinScheduler)
    "properties": {
      "endpoint": "http://localhost:12345", // Endpoint
      "project": "Project1,Project2", // Project name
      "token": "***********************" // Token
    },
    "operaterType": "AUTO" // Connection type (AUTO: Automatically get scheduling information through the API)
  },
  "conf": {

  }
}

For batch discovery, `-f` and `-o` must be folder paths. The tool creates a separate export package per project.

sh ./bin/run.sh read \
-c ./conf/<your_config_file>.json \
-f ./data/0_OriginalPackage/ \
-o ./data/1_ReaderOutput/ \
-t <dolphinv1/2/3-reader>

5.2. (Manual mode) What if there is no API?

If the DolphinScheduler API is unavailable, create the raw information package manually in `./data/0_OriginalPackage/` and set `operaterType` to `MANUAL`. The tool uses this package to complete discovery.

Delete the comments from the JSON file before use.

{
  "schedule_datasource": {
    "name": "YourDolphin", // Give your DolphinScheduler data source a name.
    "type": "DolphinScheduler", // Data source type (DolphinScheduler)
    "properties": {
      "endpoint": "http://localhost:12345", // Endpoint
      "project": "Comprehensive Test", // Project name
      "token": "***********************" // Token
    },
    "operaterType": "MANUAL" // Connection type (MANUAL: Offline mode)
  },
  "conf": {

  }
}

Example of the raw package structure:

.
├── package_info.json
├── projects.json
├── projects
│   └── Comprehensive Test
│       └── processDefinition
│           └── process_definitions_page_1.json
├── datasource
│   └── datasource_page_1.json
├── resource
│   └── resources.json
└── udfFunction
    └── udf_function_page_1.json

`package_info.json` contains package information, including the DolphinScheduler version.

{
  "version": "3.2.0"
}

`projects.json` contains project information. When creating this file manually, fill in the `id`, `userId`, `code`, and `name` fields.

[
  {
    "id": 2,
    "userId": 1,
    "code": 16372996967936,
    "name": "Comprehensive Test",
    "description": "",
    "createTime": "2025-01-20 11:40:39",
    "updateTime": "2025-01-20 11:40:39",
    "perm": 0,
    "defCount": 0,
    "instRunningCount": 0
  }
]

The `projects` folder stores workflow definitions. To create it manually, name the subdirectory after the project. Export workflow definitions from Project > Workflow > Workflow Definition, rename them sequentially to `process_definitions_page_*.json`, and place them in the `processDefinition` directory.

`datasource`, `resource`, and `udfFunction` contain data source, resource file, and UDF information. DolphinScheduler lacks export features for these, so fill `datasource_page_1.json`, `resources.json`, and `udf_function_page_1.json` with an empty array `[]`. This only affects data source mapping for SQL and DataX nodes (non-custom template mode) and node-to-resource references — configure these manually in DataWorks after import.

5.3. What if the token is valid but parts of the exported workflow are missing?

First, check if the token has permissions for the project.

Some DolphinScheduler 1.x API versions may cause data loss during export. Use the statistical report to identify and recover missing workflows.

2. Convert DolphinScheduler workflows to DataWorks workflows

DataWorks fully supports DolphinScheduler scheduling capabilities. Converted workflows behave the same as in DolphinScheduler.

1. Prerequisites

The discovery tool has completed and generated the `ReaderOutput.zip` file.

(Optional, recommended) Open the export package and review the statistical report to verify the full migration scope.

2. Conversion configuration items

2.1. Conversion configuration template

Delete the comments from the JSON file before use.

{
  "conf": {},
  "self": {
    "if.use.default.convert": false,
    "if.use.migrationx.before": false,
    "if.use.dataworks.newidea": true,
    "owner.map": [ // Owner mapping
      {
        "src": "1", // DolphinScheduler user ID
        "tgt": "202006995118212119" // DataWorks user ID
      }
    ],
    "conf": [
      {
        "nodes": "all", // Scope of the rule group
        "rule": {
          "settings": {
            // Convert DolphinScheduler Shell nodes to DataWorks Shell nodes
            "workflow.converter.shellNodeType": "DIDE_SHELL",
            // Convert unknown nodes to DataWorks virtual nodes by default
            "workflow.converter.target.unknownNodeTypeAs": "VIRTUAL",
            // Convert DolphinScheduler SQL nodes to corresponding DataWorks SQL or database nodes based on the data source type
            "workflow.converter.dolphinscheduler.sqlNodeTypeMapping": {
              "CLICKHOUSE": "CLICK_SQL",
              "HIVE": "ODPS_SQL",
              "STARROCKS": "StarRocks",
              "DORIS": "HOLOGRES_SQL",
              "MYSQL": "MYSQL",
              "REDSHIFT": "Redshift",
              "SQLSERVER": "SQLSERVER",
              "PRESTO": "EMR_PRESTO",
              "POSTGRESQL": "POSTGRESQL",
              "ORACLE": "Oracle",
              "ATHENA": "MYSQL"
            },
            // Mapping of DolphinScheduler and DataWorks data source names
            "workflow.converter.connection.mapping": {
              "mysqlDb1": "dataworks_mysqlDb1",
              "srDb1": "dataworks_srDb1"
            },
            // Main compute engine attached to DataWorks (EMR, MaxCompute, or Hologres)
            "workflow.converter.target.engine.type": "EMR",
            // Convert DolphinScheduler Spark nodes to DataWorks MaxCompute Spark nodes
            "workflow.converter.sparkSubmitAs": "ODPS_SPARK",
            "workflow.converter.sparkVersion": "3.x",
          }
        }
      }
    ]
  },
  "schedule_datasource": {
    "name": "DsProject",
    "type": "DolphinScheduler"
  },
  "target_schedule_datasource": {}
}

2.2. Owner mapping

Map DolphinScheduler users to DataWorks users to preserve workflow and node ownership.

Obtain the DolphinScheduler username and ID from the user management page.

Add users as workspace members. Obtain the user ID from the upper-right corner.

You can also obtain the ID from the Owner drop-down list on the DataWorks Data Development page.

2.3. Node conversion rules

2.3.1. Scope of rules

Set the scope for node conversion rules. To apply a uniform rule to all nodes, configure `"nodes": "all"`. Typically, one `all` rule group is sufficient.

Delete the comments from the JSON file before use.

{
  "conf": {},
  "self": {
    "conf": [
      {
        "nodes": "all", // The scope of the rule group is ALL. All nodes are converted according to this rule.
        "rule": {
          "settings": {
            // Settings
          }
      }
    ]
  }
}

To apply separate rules, specify task IDs or names in `nodes`, comma-separated. Prefer IDs over names to avoid incorrect matches. Regular expressions are supported. Set a `normal` rule group as the default for remaining nodes.

Delete the comments from the JSON file before use.

{
  "conf": {},
  "self": {
    "conf": [
      {
        "nodes": "node1Name, node2Id", // The scope of the rule group is node1 and node2.
        "rule": {
          "settings": {
            // Settings 1
          }
        },
        {
        "nodes": "node3Name, node4Id", // The scope of the rule group is node3 and node4.
        "rule": {
          "settings": {
            // Settings 2
          }
        },
        {
        "nodes": "regexExpression", // Supports filtering node names with a regular expression.
        "rule": {
          "settings": {
            // Settings 3
          }
        },
        {
        "nodes": "normal", // Conversion rule for other nodes.
        "rule": {
          "settings": {
            // Settings 4
          }
        }
        ]
  }
}

2.3.2. Conversion rules

DolphinScheduler 1.x, 2.x, and 3.x support different node types, so conversion configurations vary by version.

2.3.2.1. DolphinScheduler 3.x conversion configuration items

Supported DolphinScheduler 3.x node types:

SHELL, SQL, PYTHON, DATAX, SQOOP, SEATUNNEL, HIVECLI, SPARK (Java, Python, Sql), MR, PROCEDURE, HTTP, CONDITIONS, SWITCH, DEPENDENT, and SUB_PROCESS.

Configurable DataWorks mapping rules:

SHELL (workflow.converter.shellNodeType):

Convert to DIDE_SHELL, EMR_SHELL, or VIRTUAL nodes.

SQL (workflow.converter.dolphinscheduler.sqlNodeTypeMapping):

Convert to the corresponding SQL or database nodes.

PROCEDURE (workflow.converter.dolphinscheduler.sqlNodeTypeMapping):

Convert to the corresponding SQL or database nodes.

PYTHON (workflow.converter.pyNodeType):

Convert to PYTHON, PYODPS, PYODPS3, or EMR_SHELL nodes.

HIVECLI (workflow.converter.dolphinscheduler.sqlNodeTypeMapping/HIVE):

Convert to EMR_HIVE or ODPS_SQL nodes.

SPARK (workflow.converter.sparkSubmitAs):

Convert SparkJava and SparkPython nodes to ODPS_SPARK or EMR_SPARK nodes.

Convert SparkSql nodes to ODPS_SQL or EMR_SPARK_SQL nodes.

MR (workflow.converter.mrNodeType):

Convert to ODPS_MR or EMR_MR nodes.

Node type configurations are documented in the DataWorks Specification Repository.

Node types with fixed conversion rules:

DATAX: Converted to DI nodes. Both custom template mode (JSON Script mode) and regular mode (frontend entry mode) are supported.

Supported reader plugins: MYSQL→mysql, POSTGRESQL→postgresql, ORACLE→oracle, SQLSERVER→sqlserver, ODPS→odps, OSS→oss, HIVE→hdfs, HDFS→hdfs, CLICKHOUSE→clickhouse, MONGODB→mongodb.

Supported writer plugins: MYSQL→mysql, POSTGRESQL→postgresql, ORACLE→oracle, SQLSERVER→sqlserver, ODPS→odps, OSS→oss, HIVE→hdfs, HDFS→hdfs, CLICKHOUSE→clickhouse, MONGODB→mongodb.

SQOOP: Converted to DI nodes.

Supported reader plugins: Mysql→mysql, Hive→hive, HDFS→hdfs.

Supported writer plugins: Mysql→mysql, Hive→hive, HDFS→hdfs.

SEATUNNEL: Converted to DI nodes.

Script conversion is not yet supported. Only nodes and scheduling information are converted.

HTTP: Converted to DIDE_SHELL (general Shell) nodes. The migration tool automatically concatenates the request parameters into a curl command.
SWITCH: Converted to CONTROLLER_BRANCH (branch) nodes. The functionality is the same before and after the migration.
SUB_PROCESS: Converted to SUB_PROCESS nodes. Functionality is unchanged. The migration tool enables the 'Can be referenced' switch for referenced workflows, which can then only be started by a SUB_PROCESS call and cannot be scheduled to start on their own.

DEPENDENT: Converted to VIRTUAL nodes. Dependencies become node lineage dependencies. For example, a dependency on Workflow A becomes a lineage from the tail node of Workflow A to the Dependent node. A dependency on Node A becomes a direct lineage from Node A to the Dependent node. The following figure provides an illustration.

CONDITIONS: The node contains two layers of logic, which are implemented using two-layer CONTROLLER_JOIN (merge) nodes. The CONDITIONS node has two upstream nodes (A and B) and two downstream nodes (C and D). The logical expression is `((!A&B)|(A&!B)|(!A&!B))`. If the expression is true, the flow proceeds to C. If the expression is false, the flow proceeds to D. At the upper layer, three merge nodes are generated to calculate the results of `!A&B`, `A&!B`, and `!A&!B`. At the lower layer, two nodes are generated. One node triggers the downstream node C to execute when `((!A&B)|(A&!B)|(!A&!B))==true`. The other node triggers the downstream node D to execute when `(!(!A&B)&!(A&!B)&!(!A&!B))==true`. This process replicates the effect of the CONDITIONS node.

2.3.2.2. DolphinScheduler 2.x conversion configuration items

Supported DolphinScheduler 2.x node types:

SHELL, SQL, PYTHON, DATAX, SQOOP, HIVECLI, SPARK (Java, Python, Sql), MR, PROCEDURE, HTTP, CONDITIONS, SWITCH, DEPENDENT, and SUB_PROCESS.

Same as 3.x except SEATUNNEL. Configuration is identical.

2.3.2.3. DolphinScheduler 1.x conversion configuration items

Supported DolphinScheduler 1.x node types:

SHELL, SQL, PYTHON, DATAX, SQOOP, SPARK (Java, Python, Sql), MR, CONDITIONS, DEPENDENT, and SUB_PROCESS.

Configuration is identical to 3.x.

3. Run the scheduling conversion tool

Run the conversion tool:

sh ./bin/run.sh convert \
-c ./conf/<your_config_file>.json \
-f ./data/1_ReaderOutput/<source_discovery_export_package>.zip \
-o ./data/2_ConverterOutput/<conversion_result_output_package>.zip \
-t <PluginName>

Parameters: `-c` config file, `-f` ReaderOutput path, `-o` ConverterOutput path, `-t` conversion plugin. Plugins: `dolphinv1-dw-converter` (1.x), `dolphinv2-dw-converter` (2.x), `dolphinv3-dw-converter` (3.x).

For example, to convert DolphinScheduler 3.x Project A:

sh ./bin/run.sh convert \
-c ./conf/projectA_convert.json \
-f ./data/1_ReaderOutput/projectA_ReaderOutput.zip \
-o ./data/2_ConverterOutput/projectA_ConverterOutput.zip \
-t dolphinv3-dw-converter

The tool prints progress and reports success/failure statistics on completion. Individual node failures do not block the process — fix them manually in DataWorks after migration.

4. View the conversion results

Open `ConverterOutput.zip` in `./data/2_ConverterOutput/` to preview results.

The statistical report summarizes converted workflows, nodes, resources, functions, and data sources.

The `data/project` folder is the converted scheduling migration package.

The statistical report provides two features:

1. You can edit certain workflow and node properties (shown in blue font). The import tool applies these changes automatically.

2. Delete rows in the workflow sub-table to skip workflows during import (workflow blacklist). Interdependent workflows must be imported together — separating them causes errors.

Use the overview report in scheduling migration to add or modify scheduling properties.

3. Import to DataWorks

The LHM migration tool transforms source scheduling elements into DataWorks format and imports workflows.

Multiple import rounds are supported. Workflows are automatically created or updated (overwrite mode).

1. Prerequisites

1.1. Successful conversion

The conversion tool has completed and generated the `ConverterOutput.zip` file.

(Optional, recommended) Open the conversion output package and review the statistical report to verify the full migration scope.

1.2. DataWorks configuration

Complete these steps in DataWorks:

1. Create a workspace.

2. Create an AccessKey pair with administrator permissions for the workspace. Bind it to your account for easier troubleshooting.

3. In the workspace, create data sources, attach computing resources, and create resource groups.

4. In the workspace, upload resource files and create UDFs.

1.3. Network connectivity check

Verify that you can connect to the DataWorks endpoint.

List of service endpoints:

Service endpoints

ping dataworks.aliyuncs.com

2. Import configuration items

In the `conf` folder, create a configuration file in JSON format (e.g., `writer.json`).

Delete the comments from the JSON file before use.

{
  "schedule_datasource": {
    "name": "YourDataWorks", // Give your DataWorks data source a name.
    "type": "DataWorks",
    "properties": {
      "endpoint": "dataworks.cn-hangzhou.aliyuncs.com", // Service endpoint
      "project_id": "YourProjectId", // Workspace ID
      "project_name": "YourProject", // Workspace name
      "ak": "************", // AK
      "sk": "************", // SK
    },
    "operaterType": "MANUAL"
  },
  "conf": {
    "di.resource.group.identifier": "Serverless_res_group_***_***", // Scheduling resource group
    "resource.group.identifier": "Serverless_res_group_***_***", // Data integration resource group
    "dataworks.node.type.xls": "/Software/bwm-client/conf/CodeProgramType.xls", // Path to the DataWorks node type table
    "qps.limit": 5 // QPS limit for sending API requests to DataWorks
  }
}

2.1. Service endpoint

Select a service endpoint based on your DataWorks workspace region:

Service endpoints

2.2. Workspace ID and name

In the DataWorks console, go to Workspace > Workspace Details. The workspace ID and name appear in the basic information section.

2.3. Create and grant permissions to an AccessKey pair

On the user page, create an AccessKey pair with administrator read/write permissions for the target DataWorks workspace.

Permission management involves two locations. For RAM users, first grant DataWorks operation permissions at the RAM level.

Access policy page: https://ram.console.alibabacloud.com/policies

Then, in the DataWorks workspace, assign workspace permissions to the account.

Note: If a network access control policy is set for the AccessKey, ensure the migration tool's IP address is allowed.

2.4. Resource groups

In the left navigation pane, go to the resource group page. Attach a resource group and obtain its ID.

A general-purpose resource group handles both node scheduling and data integration. You can set `resource.group.identifier` and `di.resource.group.identifier` to the same group.

2.5. QPS settings

The tool imports data through DataWorks APIs. QPS and daily call limits vary by DataWorks edition. For more information, see Limitations.

Set `"qps.limit": 5` for Basic, Standard, and Professional editions, or `"qps.limit": 20` for Enterprise Edition.

Note: Avoid running multiple import tools at the same time.

2.6. DataWorks node type ID settings

In DataWorks, some node types have different TypeIds in different regions. The specific TypeID depends on the DataWorks Data Development interface. This characteristic mainly applies to database nodes. For more information, see Database nodes.

For example, a MySQL node has a NodeTypeId of 1000039 in the Hangzhou region and 1000041 in the Shenzhen region.

The tool provides a configurable TypeId table to adapt to regional differences.

The table is imported using the import tool's configuration items:

"conf": {
    "dataworks.node.type.xls": "/Software/bwm-client/conf/CodeProgramType.xls" // Path to the DataWorks node type table
 }

To obtain the node type ID, create a workflow and node in DataWorks Data Development, then click Save to view the workflow spec.

If the node type is configured incorrectly, the following error is reported when the workflow is published.

3. Run the DataWorks import tool

Run the import tool:

sh ./bin/run.sh write \
-c ./conf/<your_config_file>.json \
-f ./data/2_ConverterOutput/<conversion_result_output_package>.zip \
-o ./data/4_WriterOutput/<import_result_storage_package>.zip \
-t dw-newide-writer

Parameters: `-c` config file path, `-f` ConverterOutput path, `-o` WriterOutput path, `-t` submission plugin name.

For example, to import Project A to DataWorks:

sh ./bin/run.sh write \
-c ./conf/projectA_write.json \
-f ./data/2_ConverterOutput/projectA_ConverterOutput.zip \
-o ./data/4_WriterOutput/projectA_WriterOutput.zip \
-t dw-newide-writer

The tool prints progress and reports success/failure statistics on completion. Individual node failures do not block the process — fix them manually in DataWorks.

4. View the import results

View imported workflows in DataWorks. To stop an in-progress import, run `jps` to find `BwmClientApp` and then `kill -9` to terminate it.

5. Q&A

5.1. The source is under continuous development. How can I submit these increments and changes to DataWorks?

The tool runs in overwrite mode. Rerun the export, conversion, and import steps to submit incremental changes. Workflows are matched by full path, so do not move them.

5.2. The source is under continuous development, and I am also modifying and managing workflows on DataWorks. Will incremental migration overwrite the changes on DataWorks?

Yes, the tool uses overwrite mode. Modify workflows in DataWorks only after migration is complete, or migrate in batches — modify each batch only after confirming it will not be overwritten.

5.3. The entire package takes too long to import. Can I import only a part of it?

Yes. In `data/project/workflow`, keep only the needed workflows, delete the rest, recompress, and run the import tool. Interdependent workflows must be imported together to preserve node lineage.