Scheduling migration features

Lakehouse Migration (LHM) provides a lightweight scheduling migration tool that exports, transforms, and imports jobs from open source or other cloud scheduling engines into DataWorks.

Key capabilities

LHM migrates jobs from open source and other cloud scheduling engines to DataWorks with the following capabilities:

Three-step process (export source jobs, transform heterogeneous jobs, import into DataWorks) with accessible intermediate results for full migration control.
Flexible transformation configurations that support multiple DataWorks compute engines, including MaxCompute, EMR, and Hologres.
Lightweight deployment that requires only JDK 17 and network connectivity.
Local execution with no data uploaded, ensuring data security.

Architecture diagram:

Migration workflow

LHM migrates jobs to DataWorks in three steps:

Export scheduling tasks from the migration source (source discovery).

The tool retrieves scheduling task information from the source and parses it into the standard LHM data structure.

Transform scheduling properties from the migration source to DataWorks properties.

The tool transforms source task properties — including task types, scheduling settings, parameters, and scripts for some task types — into DataWorks equivalents based on the standard LHM data structure.

Import scheduling tasks into DataWorks.

The tool automatically builds DataWorks workflow definitions and imports tasks via the DataWorks SDK. It automatically determines whether to create or update tasks, supporting multiple migration rounds and synchronization of source changes.

Supported scheduling engines

LHM currently supports automated migration from the following scheduling engines to DataWorks.

Open source engines

Source type	Source version	Supported node types for transformation
DolphinScheduler	1.x	Shell, SQL, Python, DataX, Sqoop, Spark (Java, Python, SQL), MapReduce, Conditions, Dependent, SubProcess
	2.x	Shell, SQL, Python, DataX, Sqoop, HiveCLI, Spark (Java, Python, SQL), MapReduce, Procedure, HTTP, Conditions, Switch, Dependent, SubProcess
	3.x	Shell, SQL, Python, DataX, Sqoop, SeaTunnel, HiveCLI, Spark (Java, Python, SQL), MapReduce, Procedure, HTTP, Conditions, Switch, Dependent, SubProcess (renamed to SubWorkflow in version 3.3.0-alpha)
Airflow	2.x	EmptyOperator, DummyOperator, ExternalTaskSensor, BashOperator, HiveToMySqlTransfer, PrestoToMySqlTransfer, PythonOperator, HiveOperator, SqoopOperator, SparkSqlOperator, SparkSubmitOperator, SQLExecuteQueryOperator, PostgresOperator, MySqlOperator
Azkaban^Beta	3.x	Noop, Shell, Subprocess
Oozie^Beta	5.x	Start, End, Kill, Decision, Fork, Join, MapReduce, Pig, FS, SubWorkflow, Java
HUE^Beta	Latest	Fork, Join, OK, Error, Sqoop, Hive, Hive2, Shell

Latest refers to the latest version as of May 2025.

Other cloud engines

Source type	Source version	Supported node types for transformation
DataArts (DGC)	Latest	CDMJob, HiveSQL, DWSSQL, DLISQL, RDSSQL, SparkSQL, Shell, DLISpark, MRSSpark, DLFSubJob, RESTAPI, Note, Dummy
WeData	Latest	Shell, HiveSql, JDBCSql, Python, SparkPy, SparkSql, Foreach, ForeachStart, ForeachEnd, Offline Sync
Azure Data Factory (ADF)^Beta	Latest	DatabricksNotebook, ExecutePipeline, Copy, Script, Wait, WebActivity, AppendVariable, Delete, DatabricksSparkJar, DatabricksSparkPython, Fail, Filter, ForEach, GetMetadata, HDInsightHive, HDInsightMapReduce, HDInsightSpark, IfCondition, Lookup, SetVariable, SqlServerStoredProcedure, Switch, Until, Validation, SparkJob

EMR Workflow

2024.03 (Latest)

Shell, SQL, Python, DataX, Sqoop, SeaTunnel, HiveCLI, Spark, ImpalaShell, RemoteShell, MapReduce, Procedure, HTTP, Conditions, Switch, Dependent, SubProcess

DataWorks like-for-like migration

Source type	Source version	Supported node types for transformation
DataWorks	New version	All nodes included in a periodically scheduled workflow
DataWorks Spec	New version	All nodes included in a periodically scheduled workflow