Migration Hub:Lakehouse Migration Center quick start

1a. Enter basic scenario information

Provide a name and description for the migration scenario.

1b. Create a source data source

Data sources fall into two categories:

1c. Create a target data source

Create the target data source using the same process described in Step 1b. For a Hive-to-MaxCompute migration, create a MaxCompute data source as the target.

1d. Associate source and target data sources

The association list displays source data sources and their relationships to target data sources.

1. In the Operation column, click Associate Target Data Source.

2. Select the target data source for the association.

3. Click Complete to save.

2.1 Cluster exploration

Configure how and when the system discovers data and scheduling clusters in the source environment.

2.2 Metadata migration and data migration

The following example uses the Hive-to-MaxCompute (Hive2MC) scenario. Metadata migration and data migration share the same configuration.

Source-to-target mapping

Specify the mapping between source Hive databases and target MaxCompute projects. To configure in bulk, click Download Template, fill in the mapping information, and upload the file.

Blacklist

Specify tables to exclude during migration. Use the format dbname.tablename for individual tables, or dbname.* to exclude all tables in a database.

Migration method

Advanced settings -- global parameters (optional)

Advanced settings -- custom parameters (optional)

2.3 Data verification

Configure verification rules to validate data integrity after migration.

1. Create a verification engine.

2. Define verification rules:

   • Left variable: Table name or data type in the table.

   • Logical condition: Equal to, contains, or does not contain.

   • Right variable: Keywords (when the left variable is a table name) or the source/target verification execution engine (when the left variable is a data type).

3. Select a verification mode: data volume (count) comparison, metric comparison, weak content comparison, custom comparison, full text comparison, or percentage of null values comparison.

4. Set the Tolerance for difference: Define acceptable variance rules. Add or remove rules as needed.

2.4 SQL code conversion

Upload a table mapping file that specifies the mapping between source and target fully qualified names. For example, map the Hive qualified name db.tablename to the MaxCompute qualified name project.tablename.

2.5 Task scheduling migration

1. Blacklist: Specify objects to skip, such as workflows or task nodes.

2. Submission method: Submit through the migration assistant, or submit through the DataWorks OpenAPI (recommended).

3. Target engine: Select the target engine information.

4. Default conversion rules: Edit the JSON-based conversion rules. The default configuration:

{
  "format": "SPEC",
  "locale": "zh_CN",
  "skipUnSupportType": true,
  "transformContinueWithError": true,
  "specContinueWithError": true,
  "tempTaskTypes": [
    "SPARK",
    "HIVECLI"
  ],
  "skipTaskCodes": [],
  "settings": {
    "workflow.converter.shellNodeType": "EMR_SHELL",
    "workflow.converter.commandSqlAs": "EMR_SQL",
    "workflow.converter.sparkSubmitAs": "EMR_SPARK",
    "workflow.converter.target.unknownNodeTypeAs": "DIDE_SHELL",
    "workflow.converter.mrNodeType": "EMR_MR",
    "workflow.converter.target.engine.type": "EMR",
    "workflow.converter.dolphinscheduler.sqlNodeTypeMapping": {
      "POSTGRESQL": "EMR_HIVE",
      "MYSQL": "EMR_HIVE",
      "HIVE": "EMR_HIVE",
      "CLICKHOUSE": "CLICK_SQL"
    }
  }
}

Specify node types that require custom processing in the tempTaskTypes array. If left empty, nodes are converted according to the default rules defined in settings. If specified, custom rules take precedence and the original content is preserved.

1. Custom conversion plugin and rules: To apply custom conversion rules, first download the conversion plugin project, customize it, and upload it.

Custom conversion rule example:

{
  "if.use.migrationx.before": false,
  "if.use.default.convert": true,
  "conf": [
    {
      "rule": {
        "settings": {
          "workflow.converter.shellNodeType": "DIDE_SHELL",
          "workflow.converter.commandSqlAs": "DIDE_SHELL",
          "workflow.converter.sparkSubmitAs": "ODPS_SPARK",
          "workflow.converter.target.unknownNodeTypeAs": "DIDE_SHELL",
          "workflow.converter.mrNodeType": "ODPS_MR",
          "workflow.converter.target.engine.type": "MaxCompute",
          "workflow.converter.dolphinscheduler.sqlNodeTypeMapping": {
            "POSTGRESQL": "ODPS_SQL",
            "MYSQL": "ODPS_SQL",
            "HIVE": "ODPS_SQL"
          }
        }
      },
      "nodes": "all, name, idsStr"
    }
  ]
}

Custom rules override default rules when conflicts exist. You can configure multiple JSON segments to apply different conversion rules to different node ranges, allowing source shell tasks of different types to convert to different target node types.

1. Submission settings: Configure status preferences for elements submitted to the target DataWorks environment.

After completing all configuration, click Save.

3.1 Start the pipeline

Click Running Status to view the end-to-end migration progress. Exploration tasks run according to the cron schedule configured in Step 2.1.

3.2 Cluster exploration

• Data cluster exploration: The Exploration Agent scans the source cluster and reports results. View exploration result details for each run.

• Scheduling cluster exploration: The agent scans scheduling workflows. View exploration result details for each run.

3.3 Data migration

The first exploration of a source data cluster automatically generates a full data migration task. Each subsequent exploration compares results with the previous run and generates incremental migration tasks for changed objects.

Each atomic object (table or partition) follows this execution sequence:

An object is considered successfully migrated only when all three phases complete and pass verification.

View migration task details to see statistics for each task. For atomic objects that fail migration, click retry to re-attempt.

3.4 Scheduling migration

Each scheduling cluster exploration generates a batch of task instances to migrate. Perform the following steps manually:

1. Run the conversion to transform source scheduling definitions into the target format.

2. Review the conversion report.

3. Submit the converted definitions to the target DataWorks environment.