Migrate Azure Data Factory Workflows to DataWorks via LHM - DataWorks - Alibaba Cloud - Migration Hub

Migrate scheduling workflows from Azure Data Factory (ADF) to DataWorks by using the LHM migration tool. The migration involves three steps: exporting tasks from ADF, converting the scheduling tasks, and importing the tasks into DataWorks.

1. Prepare the environment

1.1. Prepare the runtime environment

No.	Item	Specifications	Quantity	Notes
1	ECS	4-core 16 GB or higher	1	CentOS or AliyunOS images are supported.
2	JDK 17
3	Runtime package			Download link

1.2. Prepare the network

The VPC that contains the ECS instance must have public network access.

1.3. Prepare accounts and permissions

1.3.1. Use service registration authentication

Obtain a client secret.

1.3.2. Configure Azure Data Factory permissions

1.3.3. Configure blob permissions

This step is optional. To read file content from a blob in a node, you must grant permissions to the blob. You can use the same method that you used to grant permissions for ADF service registration.

1.3.4. Configure Databricks permissions

This step is optional. It is required only to read external script files for Databricks nodes in ADF.

Go to the Databricks workbench.
Click User Settings.
On the left, go to the Developer page.
Click Generate new token.

2. Export tasks from Azure Data Factory

You can export tasks by using an SDK. The dependencies are as follows:

<dependency>
      <groupId>com.azure.resourcemanager</groupId>
      <artifactId>azure-resourcemanager-datafactory</artifactId>
    </dependency>
    <dependency>
      <groupId>com.azure</groupId>
      <artifactId>azure-identity</artifactId>
    </dependency>
    <dependency>
      <groupId>com.azure</groupId>
      <artifactId>azure-storage-blob</artifactId>
    </dependency>
    <dependency>
      <groupId>com.databricks</groupId>
      <artifactId>databricks-sdk-java</artifactId>
    </dependency>

2.1. Configuration file

{
  "schedule_datasource": {
    "name": "name",
    "type": "adf",
    "properties": {
      "isUseProxy": false,
      "proxyHost": "proxy.msl.cn",
      "proxyPort": "8080",
      "AzureCloud": "AZURE_CHINA_CLOUD",
      "apiMode": "sdk"
      "endpoint": "https://management.azure.com",
      "subscriptionId": "xxx",
      "factory": "bigdata-adf-jiman",
      "project": "bigdata-adf-jiman",
      "resourceGroupName": "biadata",
      "tenantId": "xxx",
      "clientId": "xxx",
      "clientSecretValue": "xxxx",
      "dbr_endpoint": "https://adb-xxxxx.16.azuredatabricks.net",
      "dbr_token": "xxxxx",
      "pipelineNameWhite": "dbr-demo"
    },
    "operaterType": "AUTO"
  }
}

2.2. Parameter descriptions

No.	Parameter	Required	Example	Notes
1	isUseProxy	No	false	Required to access Azure in proxy mode.
2	proxyHost	No	proxy.msl.cn	Required to access Azure in proxy mode.
3	proxyPort	No	8080	Required to access Azure in proxy mode.
4	AzureCloud	No	AZURE_CHINA_CLOUD	Valid values: Each value corresponds to a different region. Default value: AZURE_CHINA_CLOUD. AZURE_PUBLIC_CLOUD AZURE_CHINA_CLOUD AZURE_US_GOVERNMENT_CLOUD
5	endpoint	No	https://portal.azure.com	Associated with AzureCloud. Optional.
6	subscriptionId	Yes	xxxxx	The subscription ID. You can obtain it from the overview page of ADF.
7	resourceGroupName	Yes	xxxxx	The name of the resource group. You can obtain it from the overview page of ADF.
8	factory	Yes	bigdata-adf-jiman	An identifier for the data factory.
9	project	No	bigdata-adf-jiman	Set this parameter to the same value as factory.
10	tenantId	Yes	xxxxx	Obtained during the service registration step.
11	clientId	Yes	xxxxx	Obtained during the service registration step.
12	clientSecretValue	Yes	xxxxx	Obtained during the service registration step.
13	dbr_endpoint	No	xxxxx	Obtained from the Databricks home page.
14	dbr_token	No	xxxxx	Obtained from the Databricks home page.
15	pipelineNameWhite	No	dbr-demo	An allowlist. To specify multiple pipeline names, separate them with commas (,).

2.3. Run the export command

mkdir result
sh ./bin/run.sh read \
-c ./conf/<your_config_file>.json \
-o ./data/1_ReaderOutput/<source_export_package>.zip \
-t adf-reader

Parameter descriptions

No.	Parameter	Required	Example	Notes
1	-c	Yes	./conf/<your_config_file>.json	The path of the configuration file.
2	-o	Yes	./data/1_ReaderOutput/<source_export_package>.zip	The output path.
3	-t	Yes	adf-reader	The plugin type. This is a static field.

Example of a complete command

mkdir result
sh ./bin/run.sh read -c ./conf/read.json -f ./result/temp.zip -o ./result/read_out.zip  -t adf-reader

2.4. View the export results

Open the generated ReaderOutput.zip package in the ./data/1_ReaderOutput/ directory to preview the export results. The statistical report summarizes basic information about workflows, nodes, resources, functions, and data sources. The data/project folder contains the standardized data structure for the scheduling information.

The statistical report provides two features:

1. You can change workflow and node properties in the report. Editable fields are identified by blue font. When you import the tasks into DataWorks, the tool applies these property changes.

2. You can skip specific workflows during import by deleting their corresponding rows in the workflow child table. This acts as a workflow blocklist. Note: Workflows with mutual dependencies must be imported in the same batch. Do not use the blocklist to separate them, because this causes an error.

For more information, see Use the overview report in scheduling migration to add or modify scheduling properties.

3. Convert scheduling tasks

3.1. Configuration file

{
  "conf": {
    "locale": "zh_CN"
  },
  "self": {
    "if.use.default.convert": false,
    "if.use.migrationx.before": false,
    "if.use.dataworks.newidea": true,
    "filter.rule": []
  },
  "schedule_datasource": {
    "name": "adf",
    "type": "Adf"
  },
  "target_schedule_datasource": {
    "name": "name",
    "type": "DataWorks"
  }
}

3.2. Parameter descriptions

The default settings are sufficient.

No.

Parameter

Required

Example

Notes

filter.rule

[

{

"type": "black",

"element": "node",

"field": "name",

"value": "DataXNode"

}

]

The filter rule. Optional.

type specifies a blocklist or allowlist. Valid values: BLACK and WHITE.

element specifies the element type to filter. Supported types: NODE and WORKFLOW.

field specifies whether to filter by name or ID. Valid values: ID and NAME.

value specifies the value. To specify multiple values, separate them with commas (,).

schedule_datasource.name identifies the data source, corresponding to the root directory of all workflows.

3.3. Built-in node conversion and mapping logic

3.3.1. General node mapping logic

ADF node type	DataWorks node type	Notes
Copy	DI	Currently, only the original JSON script is available. The specific DI node content depends on the data source type. Conversion requires information about all DI-related data source types.
Delete	DIDE_SHELL	A shell script that concatenates file paths for deletion.
SqlServerStoredProcedure	ODPS-SQL	Defaults to an ODPS node. If the logic determines a SQL Server source, the node type is Sql Server.
DatabricksSparkJar	ODPS-SPARK	Converted using existing parameters. Some information may be missing.
DatabricksSparkPython	ODPS-SPARK	Converted using existing parameters. Some information may be missing.
DatabricksNotebook	NOTEBOOK	Scala-related cells are not processed.
SynapseNotebook	NOTEBOOK	Scala-related cells are not processed.
SparkJob	ODPS-SPARK	Currently converted to a virtual node.
AppendVariable	CONTROLLER_ASSIGNMENT	Assignment node.
ExecutePipeline	SUB_PROCESS
Script	ODPS_SQL
Wait	DIDE_SHELL
WebActivity	DIDE_SHELL
IfCondition	CONTROLLER_BRANCH	Converted to a branch node, a subprocess, and a merge node.
FOREACH	CONTROLLER_TRAVERSE	Foreach
Switch	CONTROLLER_BRANCH	Converted to a branch node, a subprocess, and a merge node.
Until	CONTROLLER_CYCLE	do-while node.
Lookup	CONTROLLER_ASSIGNMENT
Filter	CONTROLLER_ASSIGNMENT
GetMeta	CONTROLLER_ASSIGNMENT
SetVariable	CONTROLLER_ASSIGNMENT
HDInsightHive	ODPS-SQL
HDInsightSpark	ODPS_Spark
HDInsightMapReduce	ODPS_MR
Other	VIRTUAL

Note: You must manually upload resource files.

3.3.2. Logical node mapping logic

ifcondition

foreach

switch

3.3.3. Run the conversion command

sh ./bin/run.sh convert \
-c ./conf/<your_config_file>.json \
-f ./data/1_ReaderOutput/<source_export_package>.zip \
-o ./data/2_ConverterOutput/<conversion_output_package>.zip \
-t wedata-dw-converter

No.	Parameter	Required	Example	Notes
1	-c	Yes	./conf/<your_config_file>.json	The path of the configuration file.
2	-f	Yes	./data/1_ReaderOutput/<source_export_package>.zip	The exported package from the source.
3	-o	Yes	./result/convert_out.zip	The path for the output package.
4	-t	Yes	adf-dw-converter	The plugin type. This is a static field.

3.3.4. View the conversion results

Open the generated ConverterOutput.zip package in the ./data/2_ConverterOutput/ directory to preview the conversion results.

The statistical report summarizes basic information about the converted workflows, nodes, resources, functions, and data sources.

The data/project folder contains the main converted scheduling migration package.

The statistical report provides two features:

1. You can change workflow and node properties in the report. Editable fields are identified by blue font. When you import the tasks into DataWorks, the tool applies these property changes.

For more information, see Use the overview report in scheduling migration to add or modify scheduling properties.

4. Import tasks into DataWorks

The LHM migration tool converts scheduling elements from the source to the DataWorks scheduling format and provides a unified entry point for importing workflows into DataWorks across different migration scenarios.

The import tool supports multiple write operations and automatically creates or updates workflows in overwrite mode.

1. Prerequisites

1.1. Successful conversion

The conversion tool must have run successfully, converted the source scheduling information into the DataWorks scheduling format, and generated the ConverterOutput.zip package.

(Optional but recommended) Open the conversion output package and view the statistical report to verify that the migration scope was converted successfully.

1.2. DataWorks configuration

In DataWorks, perform the following actions:

1. Create a workspace.

2. Create an AccessKey pair (AccessKey ID and AccessKey secret) and ensure that it has administrator permissions for the workspace. We recommend that you create an AccessKey pair bound to your account for easier troubleshooting.

3. In the workspace, create data sources, attach compute resources, and create resource groups.

4. In the workspace, upload resource files and create user-defined functions (UDFs).

1.3. Network connectivity check

Verify the connection to the DataWorks endpoint.

For a list of endpoints, see:

Endpoints

ping dataworks.aliyuncs.com

2. Import configuration items

Create a JSON export configuration file, such as writer.json, in the conf folder of the project.

Before you use the file, you must delete the comments from the JSON.

{
  "schedule_datasource": {
    "name": "YourDataWorks", // Give your DataWorks data source a name.
    "type": "DataWorks",
    "properties": {
      "endpoint": "dataworks.cn-hangzhou.aliyuncs.com", // Endpoint
      "project_id": "YourProjectId", // Workspace ID
      "project_name": "YourProject", // Workspace name
      "ak": "************", // AccessKey ID
      "sk": "************" // AccessKey secret
    },
    "operaterType": "MANUAL"
  },
  "conf": {
    "di.resource.group.identifier": "Serverless_res_group_***_***", // Scheduling resource group
    "resource.group.identifier": "Serverless_res_group_***_***", // Data integration resource group
    "dataworks.node.type.xls": "/Software/bwm-client/conf/CodeProgramType.xls", // Path of the DataWorks node type table
    "qps.limit": 5 // The upper limit for queries per second (QPS) of API requests sent to DataWorks
  }
}

2.1. Endpoint

Select an endpoint based on the region of your DataWorks workspace. For more information, see:

Endpoints

2.2. Workspace ID and name

Open the DataWorks console and go to the workspace details page. You can obtain the workspace ID and name from the basic information on the right.

2.3. Create and authorize an AccessKey pair

On the user page, create an AccessKey pair with administrator read and write permissions for the target DataWorks workspace.

Permission management involves two locations. If you use a RAM user, you must first grant the RAM user permissions to operate DataWorks.

Policy page: https://ram.console.alibabacloud.com/policies

Then, in the DataWorks workspace, assign workspace permissions to the account.

Note: You can set a network access control policy for the AccessKey. Ensure that the IP address of the machine where the migration tool is located is allowed to access the network.

2.4. Resource group

On the left menu of the DataWorks workspace details page, go to the resource group page. You can attach a resource group and obtain its ID.

A general-purpose resource group can be used for node scheduling and data integration. You can set the scheduling resource group (resource.group.identifier) and the data integration resource group (di.resource.group.identifier) to the same general-purpose resource group.

2.5. QPS settings

The tool imports data by calling DataWorks APIs. Different DataWorks editions have different QPS and daily call limits for read and write OpenAPI operations. For more information, see Limits.

For DataWorks Basic, Standard, and Professional editions, you can set "qps.limit" to 5. For Enterprise Edition, you can set "qps.limit" to 20.

Note: Avoid running multiple import tools at the same time.

2.6. DataWorks node type ID settings

In DataWorks, some node types have different TypeIds in different regions. The specific TypeID is determined by the Data Development interface in DataWorks. This applies mainly to database nodes. For more information, see Database nodes.

For example, the NodeTypeId for a MySQL node is 1000039 in the Hangzhou region and 1000041 in the Shenzhen region.

To accommodate these regional differences, the tool provides a configurable method for setting the node TypeId table.

The table is imported by using a configuration item in the import tool:

"conf": {
    "dataworks.node.type.xls": "/Software/bwm-client/conf/CodeProgramType.xls" // Path of the DataWorks node type table
 }

To obtain a node type ID from DataWorks Data Development, create a workflow, add a node, save the workflow, and then view the workflow Spec.

If the node type is configured incorrectly, the following error is reported when you publish the workflow.

3. Run the DataWorks import tool

The import tool is called from the command line:

sh ./bin/run.sh write \
-c ./conf/<your_config_file>.json \
-f ./data/2_ConverterOutput/<conversion_output_package>.zip \
-o ./data/4_WriterOutput/<import_result_package>.zip \
-t dw-newide-writer

In the command, -c specifies the configuration file path, -f specifies the ConverterOutput package path, -o specifies the WriterOutput package path, and -t specifies the submission plugin name.

For example, to import Project A into DataWorks:

sh ./bin/run.sh write \
-c ./conf/projectA_write.json \
-f ./data/2_ConverterOutput/projectA_ConverterOutput.zip \
-o ./data/4_WriterOutput/projectA_WriterOutput.zip \
-t dw-newide-writer

The import tool prints progress information during runtime. Check for errors during the process. After the import is complete, statistics on successful and failed imports are printed in the command line. Note that failures of individual nodes do not stop the overall import process. If a small number of nodes fail to import, you can manually modify them in DataWorks.

4. View the import results

After the import is complete, view the results in DataWorks. You can also monitor the workflow import process. If you find a problem and need to stop the import, run the jps command to find BwmClientApp and then run the kill -9 command to terminate the process.

5. Q&A

5.1. The source is under continuous development. How are these increments and changes submitted to DataWorks?

The migration tool operates in overwrite mode. You can rerun the export, conversion, and import processes to submit incremental changes from the source to DataWorks. The tool matches workflows by their full path to decide whether to create or update them. To ensure that changes are migrated correctly, do not move the workflows.

5.2. The source is under continuous development, and workflows in DataWorks are being transformed and governed. Will incremental migration overwrite the changes in DataWorks?

Yes. The migration tool operates in overwrite mode. We recommend that you perform subsequent transformations in DataWorks only after the migration is complete. Alternatively, use a phased migration approach. After a batch of workflows is migrated and you confirm that they will not be overwritten again, you can start transforming them in DataWorks. Different batches do not affect each other.

5.3. The entire package takes too long to import. Can I import only a part of it?

Yes. You can manually trim the package to perform a partial import. In the data/project/workflow folder, keep the workflows that you want to import and delete the others. Then, recompress the folder into a package and run the import tool. Note that workflows with mutual dependencies must be imported together. Otherwise, the node lineage between them is lost.