Data Integration supports offline synchronization of entire databases from sources such as AnalyticDB for MySQL 3.0, ClickHouse, Hologres, and PolarDB to a Hologres database. This topic describes how to perform a one-time offline synchronization of an entire Hologres database to another Hologres database.
Prerequisites
You have purchased a Serverless resource group or an exclusive resource group for Data Integration.
You have created Hologres data sources for the source and destination. For more information, see Data Source Configuration.
You have established a network connection between the resource group and the data sources. For more information, see Network connectivity solutions.
Procedure
1. Select a sync task type
Go to the Data Integration page.
Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose . On the page that appears, select the desired workspace from the drop-down list and click Go to Data Integration.
In the navigation pane on the left, click Sync Task. At the top of the page, click Create Sync Task to go to the task creation page. Configure the following basic information.
Data Source And Destination:
Hologres→HologresNew Task Name: Enter a custom name for the sync task.
Synchronization Type:
Entire Database Offline.Synchronization Steps: Select Full Synchronization and Incremental Synchronization.
2. Configure the network and resources
In the Network and Resource Configuration section, select the Resource Group for the sync task. You can allocate a specific number of CUs for Task Resource Usage.
For Source Data Source, select the
Hologresdata source that you added. For Destination Data Source, select theHologresdata source that you added. Then, click Test Connectivity.
After you confirm that the connections to the source and destination data sources are successful, click Next.
3. Select the databases and tables to synchronize
In the Source Databases and Tables section, select the tables that you want to synchronize from the source data source and click the 
icon to move them to the Selected Databases and Tables list on the right.
4. Configure full and incremental synchronization
Configure the full and incremental synchronization types for the task.
If you selected Full Synchronization and Incremental Synchronization for Synchronization Steps, this section defaults to one-time full synchronization and recurring incremental synchronization. You cannot change this setting.
If you selected Full Synchronization for Synchronization Steps, you can select one-time full synchronization or recurring full synchronization for the task.
If you selected Incremental Synchronization for Synchronization Steps, you can select one-time incremental synchronization or recurring incremental synchronization for the task.
NoteThe following section uses one-time full and recurring incremental synchronization as an example to describe the key configuration.
Configure the recurring schedule parameters.
If your task involves recurring synchronization, click Recurring Schedule Parameters to configure the parameters.
5. Configure destination table mapping
After you select the tables to synchronize, they are displayed on this page. By default, the properties of the destination tables are in the To Be Refreshed state. You must define and confirm the mapping between the source and destination tables to specify the data read and write relationship. Then, click Refresh Mapping to proceed. You can refresh the mapping directly or customize the rules for the destination tables before you refresh the mapping.
You can select the tables to synchronize and click Batch Refresh Mapping. If no mapping rules are configured, the default table name rule is
${Source Table Name}. If a table with the same name does not exist in the destination, a new table is automatically created.Because recurring scheduling is required, you must define the properties for the recurring scheduling task. These properties include Scheduling Cycle, Scheduling Time, and Scheduling Resource Group. The scheduling configuration for this synchronization is the same as the node scheduling configuration in Data Development. For more information about the parameters, see Node scheduling.
You must set the Incremental Condition to a WHERE clause to filter the source data. You only need to enter the clause itself, not the WHERE keyword. If you enable a recurring configuration, you can use system parameters in the clause.
In the Custom Destination Schema Name Mapping column, click the Configure button to customize the destination schema name rule.
You can combine built-in variables and manually entered strings to create the destination schema name. You can also edit the built-in variables. For example, you can create a new schema name rule that adds a suffix to the source database name to form the destination schema name.
In the Custom Destination Database Name Mapping column, click the Configure button to customize the destination database name rule.
You can combine built-in variables and manually entered strings to create the destination database name. You can also edit the built-in variables. For example, you can create a new database name rule that adds a suffix to the source database name to form the destination database name.
In the Custom Destination Table Name Mapping column, click the Edit button to customize the destination table name rule.
You can combine built-in variables and manually entered strings to create the destination table name. You can also edit the built-in variables. For example, you can create a new table name rule that adds a suffix to the source table name to form the destination table name.
1. Edit field type mapping
The sync task has a default mapping between source and destination field types. In the upper-right corner of the table, click Edit Field Type Mapping to customize the mapping between the field types of the source and destination tables. After you complete the configuration, click Apply And Refresh Mapping.
2. Edit the destination table schema and assign values to fields
If the destination table is in the To Be Created state, you can add new fields to its original table schema. Perform the following steps:
Add fields to the destination table.
Add a field to a single table: Click the
icon in the Target Table Name column to add a field.Add fields in batches: Select all tables to synchronize. At the bottom of the table, choose .
Assign values to the new fields by performing the following operations:
Assign a value to a single table: Click the Configure button in the Destination Table Field Value Assignment column.
Assign values in batches: At the bottom of the list, choose to assign values to the same fields in the destination tables in batches.
NoteYou can assign constants and variables. Click the
icon to switch between assignment modes.
3. Configure advanced parameters
To perform a fine-grained configuration for the task to meet custom synchronization requirements, click Configure in the Custom Advanced Parameters column to modify the advanced parameters.
Modify these parameters only if you fully understand their meanings to avoid unexpected errors or data quality issues.
6. Configure advanced parameters
The sync task provides parameters that you can modify as needed. For example, you can set a limit on the maximum number of connections to prevent the current synchronization solution from putting too much pressure on the database and affecting production.
Modify these parameters only if you fully understand their meanings to avoid unexpected errors or data quality issues.
In the upper-right corner of the page, click Advanced Parameter Configuration to go to the advanced parameter configuration page.
On the Advanced Parameter Configuration page, modify the parameter values.
7. Configure the resource group
In the upper-right corner of the page, click Resource Group Configuration to view and switch the resource group that is used by the current task.
8. Run the sync task
After you complete all the configurations, click Complete Configuration at the bottom of the page.
On the page, find the sync task that you created and click Start in the Operation column.
In the Task List, click the Name/ID of the task to view the details of the execution process.
9. Configure alerts
After the task runs, a scheduling task is generated in the Operation Center. To prevent data synchronization latency that is caused by task errors, you can configure an alert policy for the sync task.
In the Task List, find the running sync task. In the Operation column, choose to open the task editing page.
Click Next. In the upper-right corner of the page, click Alert Configuration to go to the alert settings page.
In the Scheduling Information column, click the generated scheduling task to go to the task details page in the Operation Center and obtain the Task ID.
In the navigation pane on the left of the Operation Center, choose to go to the rule management page.
Click Create Custom Rule to set the Rule Object, Trigger Method, and Alert Action. For more information, see Rule management.
You can search for the task ID that you obtained in Rule Object to find the target task and set an alert for it.
Sync task O&M
View the task running status
After you create a sync task, you can view the list of created sync tasks and their basic information on the Sync Task page.
In the Operation column, you can Start or Stop a sync task. In the More menu, you can select options such as Edit and View.
For a running task, you can view the basic running status in the Execution Overview section. You can also click the corresponding overview area to view execution details.
For an offline synchronization task that synchronizes an entire database from Hologres to Hologres:
If the synchronization step of your task is Full Synchronization, schema migration and full synchronization are displayed.
If the synchronization step of your task is Incremental Synchronization, schema migration and incremental synchronization are displayed.
If the synchronization steps of your task are Full Synchronization + Incremental Synchronization, schema migration, full synchronization, and incremental synchronization are displayed.
Rerun a task
Direct rerun: Click Rerun without modifying the task configuration.
Effect: Reruns a one-time task. For a recurring task, this operation updates its properties.
Rerun after modification (adding or removing tables): Edit the task to add or remove tables, and then click Complete. The operation for the task changes to Apply Update. Click Apply Update to directly trigger a rerun of the modified task.
Effect: Only the newly added tables are synchronized. Tables that were previously synchronized are not synchronized again.
Rerun after modification (modifying table name information): Edit the task to modify the destination table name or switch the destination table to be synchronized, and then click Complete. The operation for the task changes to Apply Update. Click Apply Update to directly trigger a rerun of the modified task.
Effect: Only the modified tables are synchronized. Unmodified tables are not synchronized again.