All Products
Search

This Product

    DataWorks:Offline synchronization of an entire Hologres database to MaxCompute

    Document Center

    DataWorks:Offline synchronization of an entire Hologres database to MaxCompute

    Last Updated:Oct 28, 2025

    Data Integration supports the offline synchronization of entire databases from sources such as AnalyticDB for MySQL 3.0, ClickHouse, Hologres, MySQL, and PolarDB to MaxCompute. This topic uses the offline synchronization of a Hologres database to MaxCompute as an example to describe how to perform a one-time offline synchronization of an entire database.

    Prerequisites

    Limitations

    • This feature does not support MaxCompute data sources that use tenant-level schema syntax.

    • Synchronizing source data to MaxCompute external tables is not supported.

    Procedure

    I. Select the sync task type

    1. Go to the Data Integration page.

      Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose Data Integration > Data Integration. On the page that appears, select the desired workspace from the drop-down list and click Go to Data Integration.

    2. In the navigation pane on the left, click Sync Task. At the top of the page, click Create Sync Task. Configure the basic information as follows.

      • Source and Destination: Hologres → MaxCompute

      • New Task Name: Enter a name for the sync task.

      • Sync Type: Offline Database.

      • For Sync Steps, select Full Synchronization and Incremental Synchronization.

    II. Configure the network and resources

    1. In the Network and Resource Configuration section, select the Resource Group for the sync task. You can also assign CUs under Task Resource Usage.

    2. Set Source Data Source to the Hologres data source and Destination Data Source to the MaxCompute data source. Then, click Test Connectivity.image

    3. Once you confirm that the source and destination data sources are connected, click Next.

    III. Select the databases and tables to synchronize

    In this step, you can select the tables that you want to synchronize from the source data source in the Source Database and Tables section, and click the image icon to move them to the Selected Database and Tables section on the right.

    image

    IV. Set full and incremental synchronization control

    1. Configure the synchronization mode for the synchronization task.

      • If you selected both Full Synchronization and Incremental Synchronization in the Synchronization Steps, the default mode is one-time synchronization of full data and periodic synchronization of incremental data, which cannot be changed.

      • If you selected Full Synchronization in the Synchronization Steps, you can choose whether to perform one-time synchronization of full data or periodic synchronization of full data.

      • If you selected Incremental Synchronization in the Synchronization Steps, you can choose whether to perform one-time synchronization of incremental data or periodic synchronization of incremental data.

        Note

        In this example, the synchronization mode of one-time synchronization of full data and periodic synchronization of incremental data is used.

    2. Configure the parameters for periodic scheduling for the synchronization task.

      If your task involves periodic synchronization, you can click Scheduling Parameters to configure the settings.

    V. Map to destination tables

    After you select the tables to synchronize in the previous step, the tables are automatically displayed on this page. However, the properties of the destination tables are in the Pending Refresh Mapping state by default. You need to define and confirm the mapping relationship between source tables and destination tables, which is the relationship between data reading and writing. Then, click Refresh Mapping to proceed to the next step. You can directly refresh mappings between source tables and destination tables. You can also refresh mappings between source tables and destination tables after you configure settings related to destination tables.

    Note

    • You can select the tables to synchronize and click Batch Refresh Mapping. If no mapping rule is configured, the default table naming rule is ${SourceSchemaName}_${TableName}. If a table with the same name does not exist in the destination, it is automatically created.

    • Because this is a recurring schedule, you must define its properties. These properties include Scheduling Cycle, Scheduling Time, and Scheduling Resource Group. The scheduling configuration for this sync task is the same as the node scheduling configuration in Data Development. For more information, see Node Scheduling.

    • Specify a WHERE clause for the Incremental Condition to filter the source data. Enter only the content of the clause, not the WHERE keyword. If a recurring schedule is enabled, you can use system parameter variables.

    • You can click the Edit button in the Custom Destination Table Name Mapping column to customize the destination table name rule.

      You can concatenate built-in variables and specified strings into a final destination table name. You can edit built-in variables. For example, when you create a mapping rule, you can add a suffix to a variable that indicates a source table name to form a destination table name.

    1. Edit field type mapping

    The synchronization task has default mappings between source field types and destination field types. You can click Edit Field Type Mapping in the upper-right corner of the table to customize the field type mapping relationship between source tables and destination tables. After you complete the configuration, click Apply And Refresh Mapping.

    2. Edit destination table structure and add field value assignments

    When the destination table is in the To Be Created state, you can add fields to the destination table based on the original table structure. To configure advanced parameters, perform the following operations:

    1. Add fields to destination tables.

      • Add fields to a single table: Click the image.png button in the Destination Table Name column to add fields.

      • Add fields to multiple tables: Select all the tables to synchronize and choose Batch Modify > Destination Table Structure - Batch Modify And Add Fields at the bottom of the table.

    2. Assign values to the fields. You can perform one of the following operations to assign values to the fields:

      • Assign values to fields in a single table: Click the Configure button in the Destination Table Field Value Assignment column to assign values to the fields in the destination table.

      • Assign values to fields in multiple tables: Choose Batch Modify > Destination Table Field Value Assignment at the bottom of the list to assign values to the same fields in the destination tables in batches.

      Note

      When you assign values, you can assign constants and variables. You can click the image icon to switch the assignment mode.

    3. Customize advanced parameters

    If you need to fine-tune the task to meet custom synchronization requirements, you can click Configure in the Custom Advanced Parameters column to modify the advanced parameters.

    Important

    Before you modify the configurations of advanced parameters, make sure that you understand the meanings of the parameters to prevent unexpected errors or data quality issues.

    VI. Configure advanced parameters

    You can change the values of specific parameters configured for the synchronization task based on your business requirements. For example, you can specify an appropriate value for the Maximum read connections parameter to prevent the current synchronization task from imposing excessive pressure on the source database and data production from being affected.

    Note

    To prevent unexpected errors or data quality issues, make sure that you understand the meanings of the parameters before you change the values of the parameters.

    1. Click Advanced Parameter Settings in the upper-right corner of the page to go to the advanced parameter settings page.

    2. Modify the parameter values on the Advanced Parameter Settings page.

    VII. Configure the resource group

    You can click Resource Group Configuration in the upper-right corner of the page to view and switch the resource group used by the current task.

    VIII. Run the sync task

    1. After you complete all configurations, click Complete Configuration at the bottom of the page.

    2. On the Data Integration > Synchronization Task page, find the created synchronization task and click Start in the Actions column.

    3. Click Task List and then click the Name/ID of the corresponding task to view the detailed execution procedure of the task.

    IX. Configure alerts

    To prevent the failure of the synchronization task from causing latency on business data synchronization, you can configure different alert rules for the synchronization task.

    1. Click Alert Configuration in the upper-right corner of the page to go to the alert settings page.

    2. Select the scheduling task for the synchronization table and configure alerts for it. For more information, see Alert information.

    Sync task O&M

    View the task running status

    After you create a sync task, you can view the list of created sync tasks and their basic information on the Sync Task page.

    image

    • In the Actions column, you can Start or Stop a sync task. Under More, you can also perform other operations, such as Edit and View.

    • For a running task, you can view its status in the Execution Overview section. You can also click the task's overview area to view its execution details.

      image

      For an offline synchronization task of an entire database from Hologres to MaxCompute:

      • If a task's sync step is Full Synchronization, this section displays the schema migration and full synchronization.

      • If the sync step for your task is Incremental Synchronization, the schema migration and incremental synchronization steps appear here.

      • If your task performs both Full Synchronization and Incremental Synchronization, the status for schema migration, full synchronization, and incremental synchronization is displayed here.

    Rerun the task

    • Directly rerun the task: Do not modify the task configuration and directly click Rerun.

      Effect: Rerun a one-time task or update the properties of a periodic task.

    • Modify the task and rerun the task (add or remove tables): Edit the task, add or remove tables, and click Complete. The action for the task changes to Apply Update. Click Apply Update to directly trigger the modified task to rerun.

      Effect: Only the newly added tables are synchronized. The tables that have been synchronized are not synchronized again.

    • Modify the task and rerun the task: Modify the names of the destination Hive tables or use other Hive tables for data synchronization and click Complete. The action for the task changes to Apply Update. Click Apply Update to directly trigger the modified task to rerun.

      Effect: Synchronize the modified tables. The tables that are not modified are not synchronized again.

    Data development scenarios

    If you have downstream data dependencies and need to perform data development operations, you can refer to Node scheduling to set up upstream and downstream nodes. You can view the corresponding periodic task node information in the Periodic Configuration column here.

    image