DataWorks:Export tasks from open source engines

Limits

• Limits on version: You can use the DataWorks export tool to export DolphinScheduler nodes only of DolphinScheduler 1.3.x, 2.x, and 3.x and then import the nodes into DataWorks.

• Limits on node type conversion:

  • SQL nodes: Only some types of compute engines support the conversion of SQL nodes. During the node type conversion, the syntax of SQL code is not converted and the SQL code is not modified.

  • Cron expressions: In specific scenarios, cron expressions may be pruned or cron expressions may not be supported. You must check whether the scheduling time that is configured meets your business requirements. For information about scheduling time, see Configure time properties.

  • Python nodes: DataWorks does not provide Python nodes. DataWorks can convert a Python node in DolphinScheduler into a Python file resource and a Shell node that references the Python file resource. However, issues may occur when scheduling parameters of the Python node are passed. Therefore, debugging and checks are required. For information about scheduling parameters, see Configure and use scheduling parameters.

  • Depend nodes: DataWorks cannot convert Depend nodes for which cross-cycle scheduling dependencies are configured. If cross-cycle scheduling dependencies are configured for a Depend node, DataWorks converts the dependencies into the same-cycle scheduling dependencies of the mapped auto triggered node in DataWorks. For information about how to configure same-cycle scheduling dependencies, see Configure same-cycle scheduling dependencies.

Configure task type mappings

You can perform the steps in this section to download the DataWorks export tool and configure task type mappings.

1. Download the export tool.

   Download the source code for the export tool.

2. Configure task type mappings.

   Go to the directory of the export tool and view the lib, bin, and conf directories. You need to modify mappings in the dataworks-transformer-config.json file of the conf directory.

   Parameter description:

   The following code provides the parameters that need to be configured in the dataworks-transformer-config.json file to convert a DolphinScheduler node into an object of the ODPS type:

   {
     "format": "WORKFLOW",
     "locale": "zh_CN",
     "skipUnSupportType": true,
     "transformContinueWithError": true,
     "specContinueWithError": true,
     "processFilter": {
       "releaseState": "ONLINE",
       "includeSubProcess": true
     }
     "settings": {
       "workflow.converter.shellNodeType": "DIDE_SHELL",
       "workflow.converter.commandSqlAs": "ODPS_SQL",
       "workflow.converter.sparkSubmitAs": "ODPS_SPARK",
       "workflow.converter.target.unknownNodeTypeAs": "DIDE_SHELL",
       "workflow.converter.mrNodeType": "ODPS_MR",
       "workflow.converter.target.engine.type": "ODPS",
       "workflow.converter.dolphinscheduler.sqlNodeTypeMapping": {
         "POSTGRESQL": "POSTGRESQL",
         "MYSQL": "MYSQL"
       }
     },
     "replaceMapping": [
       {
         "taskType": "SHELL",
         "desc": "$[yyyyMMdd-1]",
         "pattern": "\$\[yyyyMMdd-1\]",
         "target": "\${dt}",
         "param": "dt=$[yyyyMMdd-1]"
       },
        {
         "taskType": "PYTHON",
         "desc": "$[yyyyMMdd-1]",
         "pattern": "\$\[yyyyMMdd-1\]",
         "target": "\${dt}",
         "param": "dt=$[yyyyMMdd-1]"
       }
     ]
   }

   Parameter		Description
   format		When you migrate data, you must configure this parameter based on whether the destination workspace participates in the public preview of new-version Data Studio. If the destination workspace participates in the public preview of new-version Data Studio, you must set this parameter to WORKFLOW. If the destination workspace does not participate in the public preview of new-version Data Studio, you must set this parameter to SPEC.	Required.
   locale		The language environment. Default value: zh_CN.
   skipUnSupportType		Specifies whether to skip unsupported types during task type conversion. Valid values: true and false. If you set this parameter to true, the unsupported types are skipped. If you set this parameter to false, the conversion fails.
   transformContinueWithError		Specifies whether to continue conversion if an error occurs during task type conversion. Valid values: true and false. If you set this parameter to true, the conversion continues. If you set this parameter to false, the conversion stops.
   specContinueWithError		Specifies whether to continue conversion if task type conversion fails. Valid values: true and false. If you set this parameter to true, the conversion continues. If you set this parameter to false, the conversion stops.
   processFilter	releaseState	The filter condition that is used to process a flow whose state is ONLINE.	Filtering during task type conversion is supported. If you want to filter DolphinScheduler nodes before you import them into DataWorks, you must configure this parameter.
   	includeSubProcess	Specifies whether to process a subprocess whose state is ONLINE.
   settings	workflow.converter.shellNodeType	The object type to which a Shell node in the source system is mapped in the destination system. Example value: DIDE_SHELL.	Required.
   	workflow.converter.commandSqlAs	The type of engine that is used to run an SQL node in the destination system. Example value: ODPS_SQL.
   	workflow.converter.sparkSubmitAs	The type of engine that is used to run a node submitted by using the Spark engine in the destination system. Example value: ODPS_SPARK.
   	workflow.converter.target.unknownNodeTypeAs	The default object type to which a node of an unknown type is mapped. Example value: DIDE_SHELL.
   	workflow.converter.mrNodeType	The type of engine that is used to run a MapReduce node in the destination system. Example value: ODPS_MR.
   	workflow.converter.target.engine.type	The engine type that is used by default. Example value: ODPS.
   	workflow.converter.dolphinscheduler.sqlNodeTypeMapping	The mapping between databases of SQL nodes in DolphinScheduler to databases in the destination system. Example values: "POSTGRESQL": "POSTGRESQL" "MYSQL": "MYSQL"
   replaceMapping	taskType	The task type to which a rule can be applied. Example value: Shell or Python.	You can replace node contents that are matched based on regular expressions.
   	desc	The description. This is an informative field and does not participate in processing. You can leave this parameter empty.
   	pattern	The pattern of the regular expression that needs to be replaced. Example value: $[yyyyMMdd-1].
   	target	The destination string that is obtained after replacement. Example value: ${dt}.
   	param	The value that you want to assign to the destination string. For example, you assign the value of a code variable to a node variable, such as dt=$[yyyyMMdd-1].

Export nodes from DolphinScheduler

You can use the export tool to export a DolphinScheduler job as a ZIP file. The following code provides sample commands. When you use the following code, you must configure the parameters based on your business requirements. After you run the following command, resources in DolphinScheduler are stored in the current directory by using the file name specified by -f.

python3 bin/reader.py \
-a dolphinscheduler \
-e http://xxx.xxx.xxx.xxx:12345 \
-t {token} \
-v 1.3.9  \
-p 123456,456256 \
-f project_dp01.zip\
-sr false;