Migrate DataArts DGC Workflows to DataWorks via LHM - Migration Hub - Alibaba Cloud - Migration Hub

Migrate DataArts (DGC) scheduling workflows to DataWorks by using the LHM scheduling migration tool. The migration involves three steps: exporting DataArts (DGC) tasks, converting scheduling tasks, and importing the tasks into DataWorks.

1. Export DataArts scheduling workflows

Huawei DataArts lets you batch export jobs, scripts, and resources. Follow the instructions below to create a DataArts scheduling information package. The LHM scheduling export tool parses this package and generates a standard scheduling migration package.

1.1. Prerequisites

Prepare a JDK 17 runtime environment and a Huawei Cloud account. The account must have permissions to export the DataArts jobs, scripts, and resources that you want to migrate.

1.2. Create a DataArts scheduling information package

2.1 Structure of the DataArts scheduling information package

The scheduling information package has the following structure. It includes data development jobs (workflows), data development node scripts, resource files, and data integration scripts. Export these components from DataArts and package them together.

.
├── project.json // Basic project information
├── cdms
│   └──<cluster_name>
│       └── <cdm_***>.json // Data integration script
└── jobs
    ├── jobs
    │   └── <***>.job // Data development job (workflow)
    ├── resources
    │   └── <***>.resource // Resource file
    └── scripts
        └── <***>.script // Data development node script

1.2.2. Create basic project information

In the `project.json` file, specify the basic project information, including the project ID and name.

{
  "dataArtsWorkSpaceId": "d494af814b4b496e8b1bc7a8eee68d7a",
  "dataArtsWorkspaceName": "default"
}

1.2.3. Export data development jobs (workflow definitions)

Open the DataArts console and click Data Development: https://console.huaweicloud.com/dayu

On the Job Development page, export the jobs. After the download is complete, decompress the package to the `./jobs/jobs` directory.

2.4 Export task script

Open the DataArts console and click Data Development: https://console.huaweicloud.com/dayu

On the Script Development page, export the scripts. After the download is complete, decompress the package to the `./jobs/scripts` directory.

2.5 Export resource file

Open the DataArts console and click Data Development: https://console.huaweicloud.com/dayu

On the Resource Management page, export the resources. After the download is complete, decompress the package to the `./jobs/resources` directory.

2.6 Exporting a data integration task script

Open the DataArts console and click Data Integration: https://console.huaweicloud.com/dayu

In Cluster Management, export the data integration task scripts. After the download is complete, decompress the package to the `./cdms` directory.

1.3. Run the scheduling parsing tool

Call the parsing tool from the command line. The command is as follows:

sh ./bin/run.sh read \
-f ./data/0_OriginalPackage/<DataArts_scheduling_information_package>.zip \
-o ./data/1_ReaderOutput/<source_exploration_export_package>.zip \
-t dataartsstudio-reader

`-f` specifies the path of the input package. `-o` specifies the path where the source exploration export package is generated. `-t` specifies the name of the exploration plugin.

For example, to parse Project A:

sh ./bin/run.sh read \
-f ./data/0_OriginalPackage/projectA_DataArtsPkg.zip \
-o ./data/1_ReaderOutput/projectA_ReaderOutput.zip \
-t dataartsstudio-reader

The exploration tool prints process information during runtime. Check for any errors.

4. View export results

Open the generated `ReaderOutput.zip` package in the `./data/1_ReaderOutput/` directory to preview the parsing results.

The statistical report provides a summary of the basic information about workflows, nodes, resources, functions, and data sources in the DataArts export package.

The `data/project` folder contains the standardized result of the DataArts scheduling information.

The statistical report provides two special features:

1. Some properties of workflows and nodes in the report can be modified. The modifiable fields are shown in blue font. In the next stage, scheduling conversion, the tool retrieves the property changes from the table and applies them during initialization.

2. The report lets you skip workflows during conversion by deleting rows in the workflow child table (workflow blacklist). Note: If workflows have dependencies on each other, the related workflows must be converted in the same batch. Do not separate them using the blacklist. Separating them will cause exceptions.

For more information, see Use the overview report in scheduling migration to supplement and modify scheduling properties.

2. Convert DataArts workflows to DataWorks workflows

1 Prerequisites

The exploration tool has run successfully. The DataArts scheduling information is exported, and the `ReaderOutput.zip` package is generated.

(Optional, recommended) Open the exploration export package and view the statistical report to verify that the scope of the migration has been fully exported.

2 Transform configuration items

2.1 Transform a configuration item template

Before you use the template, delete the comments in the JSON file.

{
  "name": "dataartsarts-dw-conveter",
  "self": {
    "if.use.migrationx.before": false,
    "if.use.default.convert": false,
    "if.use.dataworks.newidea": true,
    "param.map": {
      "TODAY": "$[yyyy-mm-dd]",
      "YYYY_MM_DD_1HOUR": "$[yyyy-mm-dd-hh24-1-1/24]",
      "DT": "${yyyy-mm-dd}",
      "DT_HH": "$[yyyy-mm-dd-hh24-1-1/24]",
      "DT_PRE_DAY": "$[yyyy-mm-dd-7]",
      "DT_PRE_BY_DAYS": "",
      "YESTERDAY": "${yyyy-mm-dd}",
      "DT_PRE_2DAYS": "$[yyyy-mm-dd-2]",
      "DT_PRE_3DAYS": "$[yyyy-mm-dd-3]",
      "DT_PRE_4DAYS": "$[yyyy-mm-dd-4]",
      "DT_PRE_5DAYS": "$[yyyy-mm-dd-5]",
      "YYYY_MM_DD_HOUR": "$[yyyy-mm-dd-hh24]"
    },
    "param.value.map": {
      "#{Job.getYesterday(\"yyyy-MM-dd\")}": "$[yyyy-mm-dd-1]",
      "#{DateUtil.format(DateUtil.addDays(Job.planTime,-1),\"yyyy-MM-dd\")}": "$[yyyy-mm-dd-1]",
      "#{DateUtil.format(DateUtil.addDays(Job.planTime,-8),\"yyyyMMddHH\")}": "$[yyyymmddhh24-8]",
      "#{DateUtil.format(DateUtil.addDays(Job.planTime,-1),\"yyyyMMdd\")}": "$[yyyymmdd-1]",
      "#{DateUtil.format(DateUtil.addDays(Job.planTime,-100),\"yyyy-MM-dd\")}": "$[yyyy-mm-dd-100]",
      "#{DateUtil.format(DateUtil.addDays(Job.planTime,-185),\"yyyy-MM-dd\")}": "$[yyyy-mm-dd-185]",
      "#{DateUtil.format(DateUtil.addDays(Job.planTime,0),\"yyyy-MM-dd\")}": "$[yyyy-mm-dd]",
      "#{DateUtil.format(DateUtil.addDays(Job.planTime,-2),\"yyyyMMdd\")}": "$[yyyymmdd-2]",
      "#{DateUtil.format(DateUtil.addDays(Job.planTime,-31),\"yyyy-MM-dd\")}": "$[yyyy-mm-dd-31]",
      "#{DateUtil.format(DateUtil.addDays(Job.planTime,0),\"yyyyMMddHH\")}": "$[yyyymmddhh24]",
      "#{DateUtil.format(DateUtil.addDays(Job.planTime,0),\"yyyyMMdd\")}": "$[yyyymmdd]",
      "#{DateUtil.format(DateUtil.addDays(Job.planTime,-0),\"yyyy-MM-dd\")}": "$[yyyy-mm-dd]",
      "#{DateUtil.format(DateUtil.addDays(Job.planTime,-62),\"yyyy-MM-dd\")}": "$[yyyy-mm-dd-62]",
      "#{DateUtil.format(DateUtil.addDays(Job.planTime,-2),\"yyyy-MM-dd\")}": "$[yyyy-mm-dd-2]"
    },
    "di.datasource.map": [
      {
        "dataArtsLinkDataSourceType": "DWS",
        "dataArtsLinkName": "dws_source1",
        "dwDataSourceType": "HOLOGRES",
        "dwConnectionName": "hologres_source1"
      },
      {
        "dataArtsLinkDataSourceType": "MYSQL",
        "dataArtsLinkName": "mysql_source1",
        "dwDataSourceType": "MYSQL",
        "dwConnectionName": "mysql_source1"
      },
      {
        "dataArtsLinkDataSourceType": "HIVE",
        "dataArtsLinkName": "hive_source1.db1",
        "dwDataSourceType": "ODPS",
        "dwConnectionName": "odps_source1_1"
      }
    ],
    "table.meta.list": [
      {
        "tableName": "",
        "partition": ""
      }
    ],
    "hBasePluginVersion": ""
  },
  "schedule_datasource": {},
  "target_schedule_datasource": {}
}

2.2 Scheduling parameter transform configuration item

DataArts provides many expressions, such as time expressions, that can be used in node scheduling parameters. The syntax of DataArts expressions differs from that of DataWorks expressions, and you must handle these differences during scheduling conversion. For more information, see Expression Overview_Data Governance Center DataArts Studio_Huawei Cloud and Date and Time Patterns_Data Governance Center DataArts Studio_Huawei Cloud.

The scheduling conversion configuration provides two replacement patterns.

1. `param.map`: Replaces the value of a specified variable by name. For example, if you configure `"TODAY": "$[yyyy-mm-dd]"`, the conversion tool changes the value of all scheduling variables named `TODAY` to `$[yyyy-mm-dd]`.

2. `param.value.map`: Replaces a variable value with a new one. For example, if you configure `"#{Job.getYesterday(\"yyyy-MM-dd\")}": "$[yyyy-mm-dd-1]"`, the tool finds all variables with the value `#{Job.getYesterday("yyyy-MM-dd")}` and changes their value to `$[yyyy-mm-dd-1]`.

The following example provides common mapping rules for reference.

{
  "self": {
    "param.map": {
      "TODAY": "$[yyyy-mm-dd]",
      "YYYY_MM_DD_1HOUR": "$[yyyy-mm-dd-hh24-1-1/24]",
      "DT": "${yyyy-mm-dd}",
      "DT_HH": "$[yyyy-mm-dd-hh24-1-1/24]",
      "DT_PRE_DAY": "$[yyyy-mm-dd-7]",
      "DT_PRE_BY_DAYS": "",
      "YESTERDAY": "${yyyy-mm-dd}",
      "DT_PRE_2DAYS": "$[yyyy-mm-dd-2]",
      "DT_PRE_3DAYS": "$[yyyy-mm-dd-3]",
      "DT_PRE_4DAYS": "$[yyyy-mm-dd-4]",
      "DT_PRE_5DAYS": "$[yyyy-mm-dd-5]",
      "YYYY_MM_DD_HOUR": "$[yyyy-mm-dd-hh24]"
    },
    "param.value.map": {
      "#{Job.getYesterday(\"yyyy-MM-dd\")}": "$[yyyy-mm-dd-1]",
      "#{DateUtil.format(DateUtil.addDays(Job.planTime,-1),\"yyyy-MM-dd\")}": "$[yyyy-mm-dd-1]",
      "#{DateUtil.format(DateUtil.addDays(Job.planTime,-8),\"yyyyMMddHH\")}": "$[yyyymmddhh24-8]",
      "#{DateUtil.format(DateUtil.addDays(Job.planTime,-1),\"yyyyMMdd\")}": "$[yyyymmdd-1]",
      "#{DateUtil.format(DateUtil.addDays(Job.planTime,-100),\"yyyy-MM-dd\")}": "$[yyyy-mm-dd-100]",
      "#{DateUtil.format(DateUtil.addDays(Job.planTime,-185),\"yyyy-MM-dd\")}": "$[yyyy-mm-dd-185]",
      "#{DateUtil.format(DateUtil.addDays(Job.planTime,0),\"yyyy-MM-dd\")}": "$[yyyy-mm-dd]",
      "#{DateUtil.format(DateUtil.addDays(Job.planTime,-2),\"yyyyMMdd\")}": "$[yyyymmdd-2]",
      "#{DateUtil.format(DateUtil.addDays(Job.planTime,-31),\"yyyy-MM-dd\")}": "$[yyyy-mm-dd-31]",
      "#{DateUtil.format(DateUtil.addDays(Job.planTime,0),\"yyyyMMddHH\")}": "$[yyyymmddhh24]",
      "#{DateUtil.format(DateUtil.addDays(Job.planTime,0),\"yyyyMMdd\")}": "$[yyyymmdd]",
      "#{DateUtil.format(DateUtil.addDays(Job.planTime,-0),\"yyyy-MM-dd\")}": "$[yyyy-mm-dd]",
      "#{DateUtil.format(DateUtil.addDays(Job.planTime,-62),\"yyyy-MM-dd\")}": "$[yyyy-mm-dd-62]",
      "#{DateUtil.format(DateUtil.addDays(Job.planTime,-2),\"yyyy-MM-dd\")}": "$[yyyy-mm-dd-2]"
    }
  }

2.3 Data Integration Transform Configuration Items

The data integration configuration items specify the mapping rules for data sources before and after migration. For example, a DataArts DWS data source is mapped to a DataWorks Hologres data source, and a DataArts Hive data source is mapped to a DataWorks ODPS (MaxCompute) data source. The configuration items are as follows:

`di.datasource.map`: The mapping table for DataArts and DataWorks data sources. `dataArtsLinkName` and `dataArtsLinkDataSourceType` are the name and type of the data source in DataArts. `dwConnectionName` and `dwDataSourceType` are the name and type of the data source in DataWorks.

Hive data sources are special. In a Hive-to-MaxCompute migration scenario, each database in the Hive data source usually corresponds to a workspace in MaxCompute. Therefore, you must add the database information to `dataArtsLinkName` for Hive data sources, as shown in the example below.
`table.meta.list`: After some data sources are migrated, you must add partition information for operations on specific tables. This parameter currently takes effect only when writing to a Hologres data source.
`hBasePluginVersion`: The Hbase version. For possible values, see HBase data source.

{
  "self": {
    "di.datasource.map": [
      {
        "dataArtsLinkDataSourceType": "DWS",
        "dataArtsLinkName": "dws_source1",
        "dwDataSourceType": "HOLOGRES",
        "dwConnectionName": "hologres_source1"
      },
      {
        "dataArtsLinkDataSourceType": "MYSQL",
        "dataArtsLinkName": "mysql_source1",
        "dwDataSourceType": "MYSQL",
        "dwConnectionName": "mysql_source1"
      },
      {
        "dataArtsLinkDataSourceType": "HIVE",
        "dataArtsLinkName": "hive_source1.db1",
        "dwDataSourceType": "ODPS",
        "dwConnectionName": "odps_source1_1"
      }
    ],
    "table.meta.list": [
      {
        "tableName": "",
        "partition": ""
      }
    ],
    "hBasePluginVersion": ""
  }
}

2.4 Node transform rule

The tool currently supports the conversion of the following DataArts node types:

· CDMJob, HiveSQL, DWSSQL, DLISQL, RDSSQL, and SparkSQL

· Shell, DLISpark, MRSSpark, DLFSubJob, RESTAPI, Note, and Dummy

The current conversion rules are fixed and cannot be configured. To customize the rules, contact the LHM migration tool team:

· `CDMJob (data integration)`: Converted to a DataWorks DI (data integration) task by default.

The data integration script (configuration items for data integration reader and writer plugins) is converted based on `di.datasource.map`. The tool covers the conversion of some CDM-to-DI configuration items related to data sources. In `dwDataSourceType`, enter the type of the DataWorks data source. The supported types include ODPS, ELASTICSEARCH, HBASE, HIVE, HOLOGRES, KAFKA, MONGODB, MYSQL, ORACLE, OSS, POSTGRESQL, and SQLSERVER.

· `HiveSQL`: Converted to ODPS_SQL (MaxCompute) by default.

· `DWSSQL`: Converted to ODPS_SQL (MaxCompute) by default.

· `DLISQL`: Converted to ODPS_SQL (MaxCompute) by default.

· `RDSSQL`: Converted to ODPS_SQL (MaxCompute) by default.

· `SparkSQL`: Converted to ODPS_SQL (MaxCompute) by default.

· `Shell`: Converted to DIDE_SHELL (general Shell) by default.

· `DLISpark`: Converted to ODPS_SPARK by default, with partial parameter mapping.

· `MRSSpark`: Converted to ODPS_SPARK by default, with partial parameter mapping.

· `DLFSubJob`: Converted to a SUB_PROCESS node by default. The reference relationships are correctly obtained.

· `RESTAPI`: Converted to DIDE_SHELL by default. The HTTP request in the node is implemented by constructing a `curl` command. Currently, only GET and POST requests are supported.

· `Note`: Converted to a VIRTUAL (zero load) node by default. The content is retained.

· `Dummy`: Converted to a VIRTUAL (zero load) node by default. The content is retained.

· Other unsupported node types are converted to VIRTUAL nodes.

For more information about DataWorks node types, see this enumeration class:

https://github.com/aliyun/dataworks-spec/blob/b0f4a4fd769215d5f81c0bbe990addd7498df5f4/spec/src/main/java/com/aliyun/dataworks/common/spec/domain/dw/types/CodeProgramType.java#L180

3 Running the scheduling transform tool

Call the conversion tool from the command line. The command is as follows:

sh ./bin/run.sh convert \
-c ./conf/<your_configuration_file>.json \
-f ./data/1_ReaderOutput/<source_exploration_export_package>.zip \
-o ./data/2_ConverterOutput/<conversion_result_output_package>.zip \
-t dataartsstudio-dw-converter

`-c` specifies the path of the configuration file. `-f` specifies the path where the `ReaderOutput` package is stored. `-o` specifies the path where the `ConverterOutput` package is stored. `-t` specifies the name of the conversion plugin.

For example, to convert DataArts Project A:

sh ./bin/run.sh convert \
-c ./conf/projectA_convert.json \
-f ./data/1_ReaderOutput/projectA_ReaderOutput.zip \
-o ./data/2_ConverterOutput/projectA_ConverterOutput.zip \
-t dataartsstudio-dw-converter

The conversion tool prints process information during runtime. Check for any errors. After the conversion is complete, statistics on successful and failed conversions are printed in the command line. Note that the failure to convert some nodes does not affect the overall conversion process. If a few nodes fail to be converted, you can manually modify them after migrating to DataWorks.

2.4. View the conversion results

Open the generated `ConverterOutput.zip` package in the `./data/2_ConverterOutput/` directory to preview the export results.

The statistical report provides a summary of the basic information about the converted workflows, nodes, resources, functions, and data sources.

The `data/project` folder contains the converted scheduling migration package.

The statistical report provides two special features:

1. Some properties of workflows and nodes in the report can be modified. The modifiable fields are shown in blue font. In the next stage, when importing to DataWorks, the tool retrieves the property changes from the table and applies them.

2. The report lets you skip workflows when importing to DataWorks by deleting rows in the workflow child table (workflow blacklist). Note: If workflows have dependencies on each other, the related workflows must be imported in the same batch. Do not separate them using the blacklist. Separating them will cause exceptions.

For more information, see Use the overview report in scheduling migration to supplement and modify scheduling properties.

3. Import to DataWorks

The LHM migration tool has converted the scheduling elements of the source to the DataWorks scheduling format. The tool provides a unified upload entry for different migration scenarios to import workflows into DataWorks.

The import tool supports multiple rounds of writing and automatically creates or updates workflows (overwrite mode).

1. Prerequisites

1.1 Transformation successful

The conversion tool has run successfully. The source scheduling information is converted to DataWorks scheduling information, and the `ConverterOutput.zip` package is generated.

(Optional, recommended) Open the conversion output package and view the statistical report to verify that the scope of the migration has been successfully converted.

1.2 DataWorks configuration

Perform the following actions in DataWorks:

1. Create a workspace.

2. Create an AccessKey pair (AK and SK) and ensure that it has administrative permissions on the workspace. We recommend that you create an AccessKey pair that is bound to your account to facilitate troubleshooting if write issues occur.

3. In the workspace, create data sources, attach compute resources, and create resource groups.

4. In the workspace, upload resource files and create user-defined functions (UDFs).

1.3 Network connectivity check

Verify the connection to the DataWorks endpoint.

List of endpoints:

Endpoints

ping dataworks.aliyuncs.com

2 Import configuration items

In the `conf` folder of the project directory, create an export configuration file in JSON format, such as `writer.json`.

Before you use the file, delete the comments in the JSON file.

{
  "schedule_datasource": {
    "name": "YourDataWorks", // Give your DataWorks data source a name.
    "type": "DataWorks",
    "properties": {
      "endpoint": "dataworks.cn-hangzhou.aliyuncs.com", // Endpoint
      "project_id": "YourProjectId", // Workspace ID
      "project_name": "YourProject", // Workspace name
      "ak": "************", // AK
      "sk": "************" // SK
    },
    "operaterType": "MANUAL"
  },
  "conf": {
    "di.resource.group.identifier": "Serverless_res_group_***_***", // Data integration resource group
    "resource.group.identifier": "Serverless_res_group_***_***", // Scheduling resource group
    "dataworks.node.type.xls": "/Software/bwm-client/conf/CodeProgramType.xls", // Path of the DataWorks node type table
    "qps.limit": 5 // QPS limit for sending API requests to DataWorks
  }
}

2.1 Endpoints

Select the endpoint based on the DataWorks region. For reference, see the following documentation:

Endpoint

2.2 Workspace ID and Name

Open the DataWorks console and go to the workspace details page. Get the workspace ID and name from the Basic Information section on the right.

2.3 Create an AccessKey pair and grant permissions

On the user page, create an AccessKey pair with administrative read and write permissions on the target DataWorks workspace.

Permission management involves two places. If the account is a Resource Access Management (RAM) user, you must first grant the RAM user the required permissions for DataWorks operations.

Access policy page: https://ram.console.alibabacloud.com/policies

Then, in the DataWorks workspace, grant workspace permissions to the account.

Note: You can configure a network access control policy for the AccessKey. Make sure that the IP address of the machine running the migration tool is allowed to access DataWorks.

2.4 Resource group

On the DataWorks workspace details page, go to the Resource Groups page from the menu bar on the left. Attach a resource group and get its ID.

A general-purpose resource group can be used for both node scheduling and data integration. You can set the scheduling resource group (`resource.group.identifier`) and the data integration resource group (`di.resource.group.identifier`) to the same general-purpose resource group in the configuration items.

2.5 QPS settings

The tool imports data by calling DataWorks APIs. Different DataWorks editions have different queries per second (QPS) limits and daily call limits for read and write API operations. For more information, see Limits.

For DataWorks Basic Edition, Standard Edition, and Professional Edition, we recommend that you set `"qps.limit"` to `5`. For Enterprise Edition, we recommend that you set `"qps.limit"` to `20`.

Note: Avoid running multiple import tools at the same time.

2.6 DataWorks node type ID setting

In DataWorks, some node types have different type IDs in different regions. The specific type ID depends on the actual Data Studio interface in DataWorks. This mainly applies to database nodes. For more information, see Database nodes.

For example, the `NodeTypeId` for a MySQL node is 1000039 in the China (Hangzhou) region and 1000041 in the China (Shenzhen) region.

To adapt to these regional differences, the tool lets you configure the node type ID table.

The table is specified in the import tool configuration items:

"conf": {
    "dataworks.node.type.xls": "/Software/bwm-client/conf/CodeProgramType.xls" // Path of the DataWorks node type table
 }

To obtain the node type ID from the DataWorks Data Studio interface, create a new workflow, add a node to it, and then click Save. View the spec of the workflow.

If the node type is configured incorrectly, the following error appears when the workflow is published.

3 Run the DataWorks import tool

Call the import tool from the command line. The command is as follows:

sh ./bin/run.sh write \
-c ./conf/<your_configuration_file>.json \
-f ./data/2_ConverterOutput/<conversion_result_output_package>.zip \
-o ./data/4_WriterOutput/<import_result_storage_package>.zip \
-t dw-newide-writer

`-c` specifies the path of the configuration file. `-f` specifies the path where the `ConverterOutput` package is stored. `-o` specifies the path where the `WriterOutput` package is stored. `-t` specifies the name of the submission plugin.

For example, to import Project A to DataWorks:

sh ./bin/run.sh write \
-c ./conf/projectA_write.json \
-f ./data/2_ConverterOutput/projectA_ConverterOutput.zip \
-o ./data/4_WriterOutput/projectA_WriterOutput.zip \
-t dw-newide-writer

The import tool prints process information during runtime. Check for any errors. After the import is complete, statistics on successful and failed imports are printed in the command line. Note that the failure to import some nodes does not affect the overall import process. If a few nodes fail to be imported, you can manually modify them in DataWorks.

4 View import results

After the import is complete, you can view the results in DataWorks. You can also view the process of individual workflows being imported. If you find a problem and need to stop the import, run the `jps` command to find `BwmClientApp` and then run `kill -9` to stop the import.

5 Q&A

5.1. The source is under continuous development. How do I submit these increments and changes to DataWorks?

The migration tool works in overwrite mode. Rerun the export, conversion, and import processes to submit the source increments to DataWorks. The tool matches workflows by their full path to decide whether to create or update them. To migrate changes, do not move the workflows.

5.2. The source is under continuous development, and I am also transforming and governing workflows in DataWorks. Will incremental migration overwrite the changes in DataWorks?

Yes. The migration tool works in overwrite mode. We recommend that you perform subsequent transformations in DataWorks after the migration is complete. Alternatively, use batch migration. After you confirm that the migrated workflows will not be overwritten again, you can start the transformation in DataWorks. Different batches do not affect each other.

5.3. The entire package takes too long to import. Can I import only a part of it?

Yes. You can manually trim the import package. In the `data/project/workflow` folder, keep only the workflows you want to import and delete the others. Recompress the folder into a package and then run the import tool. Workflows with mutual dependencies must be imported together. Otherwise, the node lineage between the workflows will be lost.