MaxCompute:Create a migration job

Migration job types

MMS supports the following types of migration jobs:

• Full database migration: migrates a single database.

• Partial migration (multiple tables): migrates one or more tables.

• Partial migration (multiple partitions): migrates one or more partitions.

Migration rules

• By default, destination projects, schemas, and table names are automatically mapped. To modify mappings, click Modify in the Actions column for the target database, or click the database name to open the table list and click Modify for a specific table. Default rules:

  • Destination project: Defaults to the project configured for the data source.

  • Schema: Defaults to the source database name.

  • Table name: Defaults to the source table name.

• A migration job migrates only tables and partitions not in the "Completed" state. View and modify the migration status in the Status column of the table or partition list.

Procedure

Before you begin, configure a data source.

1. Log in to the MaxCompute console and select a region in the upper-left corner.

2. In the left-side navigation pane, choose Data Transfer > Migration Service.

3. On the Migration Service page, click the Data Source tab.

   In the Actions column for the target data source, click Migration Management to view the databases.

4. In the Actions column of the target database, click Full DB Migrate or Partial Migration, and then configure the parameters.

   • If you select Full DB Migrate, you can configure a blacklist to exclude specific tables.

   • If you select Partial Migration, you go to the database page. On the Table List and Partitions tabs, select the tables or partitions to migrate, then click Create Migration Job.

   • You can also enable scheduled migration. When enabled, MMS runs a daily migration job on your schedule, migrating only tables or partitions in the "Unmigrated" state. View migration status in the Status column of the table or partition list.

   Note

   If the data source is MaxCompute

   • On the Migration Management page, click Migrate Project Properties in the upper-right corner to migrate MaxCompute project properties.

     This copies all parameters from the source project to the destination project.

   • To migrate Resources and UDFs, click Full Migration in the Full DB Migrate column, then select full database migration, Resource, or UDF as the migration object.

   Parameters

   Parameter	Required	Description
   Database Name	Yes	The database to migrate. Auto-populated.
   Job name	Yes	The migration job name.
   Data source	Yes	The current data source. Auto-populated.
   Source database	Yes	The database to migrate. Auto-populated with the current database name.
   Destination MaxCompute Project	Yes	Select the destination MaxCompute project. Note If the data source is BigQuery, the destination MaxCompute project must support schemas.
   Destination schema	Yes	Select a schema in the destination MaxCompute project. Schema operations. Note This parameter is available only for projects with the schema feature enabled.
   Table Blacklist and Whitelist	No	Select a Whitelist or Blacklist. Whitelist: Tables to migrate. Separate multiple names with commas (,). Note For a partial migration (multiple tables), the whitelist is pre-filled with the tables you selected. For a full database migration, the whitelist is empty by default. Blacklist: Tables to exclude. Separate multiple names with commas (,). Not required for partial migration (multiple partitions).
   ETA	No	Expected completion time. MMS uses this to prioritize jobs.
   Enable Check	No	When enabled, MMS verifies data integrity by comparing row counts between source and destination partitions using SELECT COUNT(*).
   Incremental Migration	No	When enabled, the job skips partitions already migrated in previous runs.
   Migrate Only Schemas	No	Creates table and partition structures in the destination project without migrating data.
   Partition Filter	No	A filter expression for partitions. Example: p1 >= '2022-03-04' and (p2 = 10 or p3 > 20) and p4 in ('abc', 'cde'). The expression supports the following syntax: p1, p2, and p3 are partition column names. Partition values can be strings or numbers. String values must be enclosed in single (') or double (") quotation marks. Values for non-INT and non-BIGINT columns can only be specified as strings. Comparison operators: >, >=, = , <, <=, <>. The IN operator is supported. Logical operators: AND and OR. Parentheses () can group conditions.
   Table Name Mapping	No	The new table name in the destination project.
   Field Name Mapping	No	The new column name in the destination project.
   Other Configurations	No	Table type mappings in JSON format. Example: { "mc.table.type": { "dataset1.table1":{"table.type":"delta","pk":["id1"]}, "dataset2.table2":{"table.type":"delta","pk":["id1","id2"]}, "dataset2.table2":{"table.type":"transactional"} } }.

5. Click Submit to create the migration job.

6. On the Migration Service page, click the Migration Job tab.

   Check the migration job status. If a job fails:

   • Click Details in the Actions column for the failed job. On the Migration Task List page, click Actions in the Log column for the corresponding task to view the failure reason.

   • Click Actions in the Retry column for the failed job to rerun it.

     Note

     Migration involves three stages: schema migration, data migration, and data validation. A failed task restarts from the data migration stage after a Retry operation.