All Products
Search

This Product

    Dataphin:Configure full database migration tasks

    Document Center

    Dataphin:Configure full database migration tasks

    Last Updated:Jun 05, 2026

    Offline full database migration synchronizes data from on-premises or ECS self-managed databases to big data services such as MaxCompute, Hive, and TDH Inceptor.

    Prerequisites

    The required source data sources are created. Full database migration supports MySQL, Microsoft SQL Server, Oracle, OceanBase, and more. For supported data sources, see Supported data sources for full database migration.

    Overview

    Unlike individual offline pipelines, full database migration lets you configure pipelines in batches and synchronize multiple tables at once.

    Procedure

    1. On the Dataphin homepage, choose Develop > Data Integration in the top navigation bar.

    2. In the top navigation bar, select the target project.

    3. In the left-side navigation pane, choose Full database migration > Offline full database migration.

    4. On the Offline full database migration page, configure the following parameters.

      1. Basic information

        Full database migration folder name: Up to 256 characters. The following characters are not supported: | / \ : ? < > * ".

      2. Data source information

        • Source

          Parameter

          Description

          Data source type

          Select the source type. For supported data sources and creation steps, see Supported data sources for full database migration.

          • Oracle data source

            • Schema: Select the schema that contains the table. Multiple schemas are supported. If unspecified, the default schema from the data source configuration is used.

            • File encoding: Required for Oracle data sources. Select a file encoding. Supported formats: UTF-8, GBK, and ISO-8859-1.

          • Microsoft SQL Server, PostgreSQL, Amazon Redshift, Amazon RDS for PostgreSQL, Amazon RDS for MySQL, Amazon RDS for SQL Server, Amazon RDS for Oracle, Amazon RDS for DB2, PolarDB-X 2.0, and GBase 8C data sources

            Schema: Select the schema that contains the table. Multiple schemas are supported. If unspecified, the default schema from the data source configuration is used.

          • DolphinDB data source

            Database: Select the database where the table resides. If blank, the database specified during data source registration is used.

          • Hive data source

            Configure the following parameters for Hive data sources.

            • File encoding: Supported formats are UTF-8 and GBK.

            • ORC table compression format: Supported formats are zlib, hadoop-snappy, lz4, and none.

            • Text table compression format: Supported formats are gzip, bzip2, lzo, lzo_deflate, hadoop_snappy, framing-snappy, zip, and zlib.

            • Parquet table compression format: Supported formats are hadoop_snappy, gzip, and lzo.

            • Field delimiter: Delimiter used when writing to the target table. Defaults to \u0001 if blank.

          Time zone

          Select a time zone matching your database configuration. The default is GMT+8 (no DST support). If your database uses a DST-aware time zone such as Asia/Shanghai, select the corresponding DST-aware time zone (for example, Asia/Shanghai) to avoid one-hour offsets in synchronized data.

          Supported time zones include: GMT+1, GMT+2, GMT+3, GMT+5:30, GMT+8, GMT+9, GMT+10, GMT-5, GMT-6, GMT-8, Africa/Cairo, America/Chicago, America/Denver, America/Los_Angeles, America/New York, America/Sao Paulo, Asia/Bangkok, Asia/Dubai, Asia/Kolkata, Asia/Shanghai, Asia/Tokyo, Atlantic/Azores, Australia/Sydney, Europe/Berlin, Europe/London, Europe/Moscow, Europe/Paris, Pacific/Auckland, and Pacific/Honolulu.

          Data source

          Select the data source, or click New Data Source to create one.

          Batch read entries

          For Oracle, Microsoft SQL Server, OceanBase, IBM DB2, PostgreSQL, Amazon Redshift, Amazon RDS for PostgreSQL, Amazon RDS for MySQL, Amazon RDS for SQL Server, Amazon RDS for Oracle, Amazon RDS for DB2, DolphinDB, and GBase 8C sources, set the number of entries per batch read. Default: 1024.

        • Target

          Parameter

          Description

          Data source type

          Select the target data source type. For supported data sources and creation steps, see Supported data sources for full database migration.

          Note

          • When synchronizing data to an AnalyticDB for PostgreSQL data source, the system creates a daily partition for the target table.

          • If you require other partitions, you can modify the prepared statement for the relevant partition in an individual pipeline after the pipelines are generated.

          Data source

          Select the target data source, or click New Data Source to create one. For supported data sources and creation steps, see Supported data sources for full database migration.

          • TDH Inceptor and ArgoDB target data source types.

            Configure the storage format.

            • TDH Inceptor: Supported storage formats are PARQUET, ORC, and TEXTFILE.

            • ArgoDB: Supported storage formats are PARQUET, ORC, TEXTFILE, and HOLODESK.

          • Hive target data source type.

            Configuration varies by data lake table format: Hudi, Iceberg, or None.

            Note

            Iceberg and Hudi are available only if the selected data source or the project's compute source has enabled and specified them.

            Hive

            • Is external table: Toggle to create an external table. Disabled by default.

            • Storage format: Supported formats are PARQUET, ORC, and TEXTFILE.

            • File encoding: When the Hive storage format is ORC, you can configure the file encoding. Supported formats are UTF-8 and GBK.

            • Compression format:

              • ORC storage format: Supported formats are zlib, hadoop-snappy, and none.

              • PARQUET storage format: Supported formats are gzip and hadoop-snappy.

              • TEXTFILE storage format: gzip, bzip2, lzo, lzo_deflate, hadoop-snappy, and zlib.

            • Performance configuration: Available when the Hive storage format is ORC. Increase the buffer size for tables with many fields to improve write performance (if memory allows), or decrease it to reduce GC overhead. Default: {"hive.exec.orc.default.buffer.size":16384} bytes. Maximum recommended: 262,144 bytes (256 KB).

            • Field delimiter: Set the field delimiter for the TEXTFILE storage format. Defaults to \u0001 if blank.

            • Delimiter handling: For the TEXTFILE storage format, define how delimiters within data are handled to prevent write errors. Options: Retain, Remove, and Replace with.

            • Row delimiter handling: For the TEXTFILE storage format, define how line breaks (\r\n or \n) within data are handled to prevent write errors. Options: Retain, Remove, and Replace with.

            • Development data source Location: For Dev-Prod projects, specify the table storage location in the DDL statement for the development data source. For example, hdfs://path_to_your_extemal_table.

            • Production data source Location: For Dev-Prod projects, specify the table storage location in the DDL statement for the production data source. For example, hdfs://path_to_your_extemal_table.

              Note

              For Basic mode projects, specify only one Location.

            Hudi

            • Execution engine: If Spark is enabled for the selected data source, choose Spark or Hive. Otherwise, only Hive is supported.

            • Hudi table type: Select MOR (merge on read) or COW (copy on write). Default: MOR.

            • Extended attributes: Enter Hudi-supported configuration properties in k=v format.

            Iceberg

            • Execution engine: If Spark is configured for the data source, Spark is selected by default. Otherwise, only Hive is available.

            • Is external table: Toggle to create an external table. Disabled by default.

            • Storage format: Supported formats are PARQUET, ORC, and TEXTFILE.

            • File encoding: When the Hive storage format is ORC, you can configure the file encoding. Supported formats are UTF-8 and GBK.

            • Compression format:

              • ORC storage format: Supported formats are zlib, hadoop-snappy, and none.

              • PARQUET storage format: Supported formats are gzip and hadoop-snappy.

              • TEXTFILE storage format: gzip, bzip2, lzo, lzo_deflate, hadoop-snappy, and zlib.

            • Performance configuration: Available when the Hive storage format is ORC. Increase the buffer size for tables with many fields to improve write performance (if memory allows), or decrease it to reduce GC overhead. Default: {"hive.exec.orc.default.buffer.size":16384} bytes. Maximum recommended: 262,144 bytes (256 KB).

            • Field delimiter: Set the field delimiter for the TEXTFILE storage format. Defaults to \u0001 if blank.

            • Delimiter handling: For the TEXTFILE storage format, define how delimiters within data are handled to prevent write errors. Options: Retain, Remove, and Replace with.

            • Row delimiter handling: For the TEXTFILE storage format, define how line breaks (\r\n or \n) within data are handled to prevent write errors. Options: Retain, Remove, and Replace with.

            • Development data source Location: For Dev-Prod projects, specify the table storage location in the DDL statement for the development data source. For example, hdfs://path_to_your_extemal_table.

            • Production data source Location: For Dev-Prod projects, specify the table storage location in the DDL statement for the production data source. For example, hdfs://path_to_your_extemal_table.

              Note

              For Basic mode projects, specify only one Location.

            Paimon

            • Execution engine: Currently, only Spark is supported.

            • Paimon table type: Select MOR (merge on read), COW (copy on write), or MOW (merge on write). Default: MOR.

            • Extended attributes: Enter Paimon-supported configuration properties in k=v format.

          • AnalyticDB for PostgreSQL, GaussDB (DWS), and OushuDB target data source types.

            Important

            The conflict resolution policy is effective only in Copy mode for PostgreSQL kernel versions later than 4.3. Use this option with caution for earlier or unknown kernel versions to avoid task failures.

            • Conflict resolution policy: When using the copy load policy, you can set a conflict resolution policy. Options include Report error on conflict and Overwrite on conflict.

            • Schema: Select the schema that contains the table. Multiple schemas are supported. If unspecified, the default schema from the data source configuration is used.

          • Lindorm target data source type.

            Configure the following parameters.

            • Storage format: Supported formats are PARQUET, ORC, TEXTFILE, and ICEBERG.

            • Compression format: Different compression formats are supported for different storage formats.

              • ORC storage format: Supported formats are zlib, hadoop-snappy, lz4, and none.

              • PARQUET storage format: Supported formats are gzip and hadoop-snappy.

              • TEXTFILE storage format: Supported formats are gzip, bzip2, lzo, lzo_deflate, hadoop-snappy, and zlib.

            • Development data source Location: For Dev-Prod projects, specify the root path for table storage in the DDL statement for the development environment. For example: /user/hive/warehouse/xxx.db.

            • Production data source Location: For Dev-Prod projects, specify the root path for table storage in the DDL statement for the production environment. For example: /user/hive/warehouse/xxx.db.

          • MaxCompute data source

            MaxCompute table type: You can select Standard table or Delta table.

          • Databricks data source

            Schema: Select the schema that contains the table. Multiple schemas are supported. If unspecified, the default schema from the data source configuration is used.

          Load policy

          • For Hive (when Data Lake Table Format is set to Hudi or Paimon), TDH Inceptor, ArgoDB, StarRocks, Oracle, MaxCompute, Lindorm (compute engine), and Data Lake Formation target data sources, the supported load policies are Overwrite data, Append data, and Update data.

            • Overwrite data: If the data to be synchronized already exists, the existing data is deleted before the new data is written.

            • Append data: Existing data is not overwritten, and the new data is appended.

            • Update data: Data is updated based on the primary key. If a primary key does not exist, the new data is inserted.

              Note

              • If the MaxCompute table type is Standard table, you can choose Append data or Overwrite data. If the MaxCompute table type is Delta table, you can choose Update data or Overwrite data.

              • When the target data source is Data Lake Formation, only Append data and Update data are supported.

          • For Hive target data sources (when Data Lake Table Format is not selected), the supported load policies are Overwrite only data written by the integration task, Append data, and Overwrite all data.

          • For AnalyticDB for PostgreSQL, GaussDB (DWS), and OushuDB target data sources, the supported load policies are insert and copy.

            • insert: Synchronizes data row by row. This method is suitable for small data volumes and improves data accuracy and integrity.

            • copy: Synchronizes data as files. This method is suitable for large data volumes and improves synchronization speed.

          Batch write data size

          For Hive (Hudi), AnalyticDB for PostgreSQL, and StarRocks targets, set the data size per batch write. The write triggers when either the size limit or entry limit is reached.

          Batch write entries

          For Hive (Hudi), AnalyticDB for PostgreSQL, and StarRocks targets, set the number of entries per batch write.

      3. Data synchronization

        • Non-FTP source

          When you select source tables, corresponding target tables are generated with the same names by default. If a Name Conversion rule is configured, converted names are used.

          Note

          If the source is an external data source without a collection task, this section is empty because metadata cannot be retrieved. Configure a collection task in Metadata Center to resolve this.

          image

          Area

          Description

          ① Operations area

          • Selected and Not Selected: You can filter source tables based on whether they are Selected or Not Selected.

          • Search for a source table: You can search for a source table by its name. The search is case-sensitive.

          • Advanced Search: The page displays up to 10,000 tables. Use Advanced Search to search tables across the entire database in the Advanced Search dialog box.

            1. Configure search method.

              You can search for sources by using Enter table names or Fuzzy search.

              • Enter table names: Enter table names in the Search content box. Use the configured delimiter to separate table names. The default delimiter is \n, which you can customize.

              • Fuzzy search: Enter keywords for table names in the Search content box. The system performs a fuzzy search based on the keywords.

            2. Enter search content.

              • The search content you enter depends on the search method.

                • Enter table names: You can enter multiple table names to search for them in batches. Use the configured delimiter to separate the names.

                • Fuzzy search: You can enter keywords from the table names to search.

            3. Search result.

              After configuring the search method and search content, click Search to display the results. In the results table, select the desired tables and choose an operation, such as Select in batches or Deselect in batches. Click OK to select or deselect the source tables.

          • Automatically delete the table that has the same name in the data source: If selected, Dataphin deletes same-named tables in the target data source before recreating them.

            Important

            If the data source is a project data source, tables with the same name in both the production and development environments will be deleted. Use this option with caution.

          • Name Conversion: Optional. Use conversion rules to replace or filter source table names and field names during synchronization.

            1. Click Name Conversion.

            2. On the Name Conversion page, configure the conversion rules.

              • Table name conversion rules: Click Create Rule. In the rule item, enter the String to be replaced in the source table name and the Replacement string in the target table name. For example, to replace datawork with dataphin in a table name, set the string to be replaced to work and the replacement string to phin.

              • Table name prefix: In the Table name prefix field, enter a prefix for the target tables. This prefix is automatically added to the beginning of the target table names during synchronization. For example, if you set the prefix to pre_ and the table name is dataphin, the target table is named pre_dataphin.

              • Table name suffix: In the Table Suffix input box, enter a suffix for the target table name. The system uses this suffix to generate the target table name during synchronization. For example, if the suffix is _prod and the table name is dataphin, the generated target table name is pre_dataphin_prod.

              • Field name rules: Click Add Rule. In the rule item, enter the String to be replaced in the source field name and the Replacement string in the target field name. For example, to replace datawork with dataphin in a field name, set the string to be replaced to work and the replacement string to phin.

            3. After configuring the rules, click OK. The Corresponding target table column will display the converted target table names.

              Note

              English letters in the replacement strings, prefixes, and suffixes are automatically converted to lowercase.

          • Check Table Name: Verifies whether the target table names already exist in the target database.

          ② Source table

          In the Source table list, select the tables to synchronize.

          ③ Corresponding target table

          A target table is generated for each selected Source table, named after the source table by default. If a Name Conversion rule is configured, the converted name is used.

          Note

          Target table names can only contain letters, numbers, and underscores. If a source table name contains other characters, you must configure a table name conversion rule.

          ④ Pipeline statistics

          The number of currently selected pipelines.

        • FTP source

          1. Click Download Excel Template, fill it out according to the instructions and upload it. To avoid parsing errors, you must strictly follow the template's format.

            Note

            You can upload a single .xlsx file or a single compressed package (which can contain one or more .xlsx files). Only the ZIP compression format is supported. The file size must be less than 50 MB.

          2. After uploading the document, click Parse File.

            image.png

            Parameter

            Description

            ① Operations area

            • Search for a source file: You can search for a source file by its name.

            • View only parsing-failed tasks: The list will only show tasks that failed to parse.

            • Automatically delete the table that has the same name in the data source: If selected, the system deletes same-named tables in the target data source before recreating them.

              Important

              If the data source is a project data source, tables with the same name in both the production and development environments will be deleted. Use this option with caution.

            • Check Table Name: Verifies whether the target table names already exist in the target database.

            ② Source file and Corresponding target table

            • Source file: In the Source file list, select the source files you want to synchronize.

            • Corresponding target table: After parsing the file, corresponding target tables are generated based on the template.

            ③ Pipeline statistics

            The number of currently selected pipelines.

      4. Task name configuration

        The Generation method sets the naming convention. Options: System Default or Custom Rule.

        Parameter

        Description

        Generation method

        System Default

        Uses the system's default naming convention.

        Custom Rule
        Important

        Before configuring a custom task name rule, you must select the source and target data sources. Otherwise, you cannot configure the rule.

        • Default rule: After selecting the source and target data sources and choosing Custom Rule, the system generates a default rule in the Task name rule text box. The default rule is ${source_data_source_type}2${target_data_source_type}_${source_table_name}.

          For example, if the source data source is MySQL, the target is Oracle, and the first source table is named source_table_name1, the default naming rule is MySQL2Oracle_${source_table_name}, and the task name preview is MySQL2Oracle_source_table_name1.

          Note

          This default rule is different from the System Default generation method.

        • Custom rule: Enter a naming rule in the Task name rule text box. You can delete the existing default rule or modify it.

          The name cannot contain the following characters: | / \ : ? < > * ". It can be up to 256 characters long. You can click on valid metadata names from the Available metadata list on the right to copy them.

          Note

          When you add metadata to the naming rule, the values in the task name preview are taken from the first table in the source table list.

      5. Synchronization and filtering

        Parameter

        Description

        Synchronization method

        Select a synchronization method: Daily Synchronization, One-time Synchronization, or Daily Synchronization + One-time Synchronization.

        • Daily Synchronization: Generates a scheduled periodic integration pipeline task for daily incremental or full data synchronization.

        • One-time Synchronization: Generates a manual integration pipeline task for one-time full synchronization of historical data.

        • Daily Synchronization + One-time Synchronization: Generates both a scheduled periodic task and a manual task for one-time full synchronization followed by daily incremental or full synchronizations.

          Note

          If the source is FTP, Daily Synchronization + One-time Synchronization is not supported.

        Create target table as

        Select the type of target table to create. Options include partitioned table and non-partitioned table. The creation rules vary based on the synchronization method:

        • Daily Synchronization: If you select partitioned table, the target table is created as a partitioned table, and data is written to the ds=${bizdate} partition by default. If you select non-partitioned table, the target table is created as a non-partitioned table.

        • One-time Synchronization: If you select partitioned table, you must configure the Partition for one-time synchronization parameter. It supports constants or partition parameters, such as the constant 20230330 or the partition parameter ds=${bizdate}. If you select non-partitioned table, the target table is created as a non-partitioned table.

        • Daily Synchronization + One-time Synchronization: The target is created as a partitioned table by default, and this cannot be changed. You must configure the Partition for one-time synchronization parameter. It supports constants or partition parameters, such as the constant 20230330 or the partition parameter ds=${bizdate}.

          Note

          Currently, one-time synchronization only supports writing data to a single specified partition of the target table. If you need to write full historical data to different partitions, you can either use an SQL task to process and write the data to the corresponding partitions after the one-time synchronization, or choose daily synchronization for incremental data and then backfill historical partitions.

        Data filtering

        • Source is not Hive or MaxCompute

          • Filter for daily synchronization: If the synchronization method includes Daily Synchronization, you can configure a filter. For example, if you set the filter to ds=${bizdate}, the task will extract all data where ds=${bizdate} from the source and write it to the specified target partition.

          • Filter for one-time synchronization: If the synchronization method includes One-time Synchronization, you can configure a filter. For example, if you set the filter to ds<${bizdate}, the task will extract all data where ds<${bizdate} from the source and write it to the specified target table or partition.

        • Source is Hive or MaxCompute

          • Partition for daily synchronization: If the source is Hive or MaxCompute, you must specify the partition to be read daily. It supports reading a single partition, such as ds=${bizdate}, or multiple partitions, such as /*query*/ds>=20230101 and ds<=20230107.

          • Partition for one-time synchronization: If the source is Hive or MaxCompute and the synchronization method includes One-time Synchronization, you must specify the partition to be read. It supports reading a single partition, such as ds=${bizdate}, or multiple partitions, such as /*query*/ds>=20230101 and ds<=20230107.

          • If partition does not exist: You can choose a policy to handle cases where the specified partition does not exist:

            • Set the task to Failed: The task is stopped and its status is set to Failed.

            • Set the task to Succeeded: The task runs successfully, but no data is written.

            • Use latest non-empty partition: For MaxCompute sources, you can use the latest non-empty partition (max_pt) of the table. If the table has no data in any partition, the task fails. This option is not supported for Hive sources.

        Note

        Data filtering is not supported if the source is FTP.

        Parameter configuration

        If the source is FTP, you can use parameters in the source file path.

      6. Scheduling and execution

        Parameter

        Description

        Scheduling configuration

        Select a scheduling configuration. Options include Concurrent scheduling and Batch scheduling.

        • Concurrent scheduling: All selected table synchronization tasks run simultaneously at 00:00 daily in the specified scheduling time zone.

        • Batch scheduling: Tasks run in batches. Set a period (0–23 hours) and up to 142 tasks per batch. For example, synchronizing 100 tables at 10 tables every 2 hours takes 20 hours. The entire cycle cannot exceed 24 hours.

        Running Timeout

        Tasks exceeding the timeout are stopped and marked as Failed. Options: System Configuration or Custom.

        • System Configuration: Uses the default timeout. Runtime settings.

        • Custom: Enter a timeout from 0 (exclusive) to 168 hours, up to two decimal places.

        Automatic retry on failure

        If a task or backfill instance fails, the system retries automatically. Set retries (0–10) and retry interval (1–60 minutes).

        Upstream dependency

        Click Add Dependency to add a physical node or logical table node as an upstream dependency. If none is configured, the tenant's virtual root node is used by default. You can also add a virtual node for scenarios like unified data backfilling.

      7. Resource configuration

        Scheduling resource group: Full database migration tasks are exclusive resource tasks. Specify a scheduling resource group for task instances. Instances consume the quota of their assigned group and enter Waiting for scheduling resources if the group has insufficient resources. Resource isolation between groups ensures scheduling stability.

        Select a resource group designated for Task Daily Scheduling and associated with the current project. Configure resource groups.

        Note

        • In Basic projects, you can configure the Scheduling resource group. In Dev-Prod projects, you can configure both the Scheduling resource group for development tasks and the Scheduling resource group for production tasks.

        • If you select Project Default Resource Group, the configuration will be automatically updated based on changes to the project's default settings.

        • By default, tasks in both development and production environments use the project's default scheduling resource group. You can change this to another resource group that is bound to the current project, including resource groups from registered scheduling clusters.

    5. After configuration, click Generate Pipelines to create the migration pipelines.

      The Run Results area shows pipeline results: source table, target table, synchronization method, task status, and remarks.

    6. After generation, a folder is created in the offline integration directory containing the pipeline tasks. Configure and publish these tasks as described in Configure offline pipeline task properties.

      If some tables fail or you need to add tables later, manually create offline pipeline or script tasks and move them into the folder.

      1. Click the image icon next to the target offline full database migration folder. Choose to create a new Offline Pipeline or Offline Script.

      2. In the Create Offline Pipeline or Create Offline Script dialog box, complete the configuration and click OK. For more information about the configuration parameters, see Create an integration task by using a single pipeline and Create an integration task by using Script Mode.

        Note

        • The created offline pipeline and script tasks will be located in the current offline full database migration folder.

        • You cannot move a folder into or out of an offline full database migration folder.

        • To move an existing offline pipeline or script task, click the image icon next to its name, select Move, and choose the target directory. You can move tasks into a full database migration folder.

        • If you delete a full database migration folder, all tasks within that folder, including offline pipeline and script tasks, are also deleted.

    Next steps

    • After publishing, view and manage integration tasks in the O&M Center.