Data Integration supports single-table real-time synchronization to Hologres from sources such as DataHub, Hologres, Kafka, and LogHub. A synchronization task reads data from a source Hologres table and writes the data to a destination table in another Hologres data source. The destination table is either automatically created based on the source table schema or an existing table that you select.
Limits
The following limits apply to single-table real-time synchronization from Hologres to Hologres:
The Hologres version must be V2.1 or later.
Incremental synchronization of Hologres partitioned tables is not supported.
Synchronization of DDL change messages of Hologres tables is not supported.
Incremental synchronization from Hologres supports the following data types:
INTEGERBIGINTTEXTCHAR(n)VARCHAR(n)REALJSONSERIALOIDINT4[]INT8[]FLOAT8[]BOOLEAN[]TEXT[]
Prerequisites
Purchase a Serverless resource group.
Create the source and destination Hologres data sources. For instructions, see Data source configuration.
Establish network connectivity between the resource group and the data sources. For details, see Overview of network connectivity solutions.
Enable Hologres binlog for the source table. For instructions, see Subscribe to Hologres binlog.
Procedure
Step 1: Select the synchronization task type
Log on to the DataWorks console. In the target region, click in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Integration.
In the left-side navigation pane, click Synchronization Task. At the top of the page, click Create Synchronization Task to open the synchronization task creation page.
Configure the following basic information:
Data source and destination: Hologres → Hologres.
New Node Name: Enter a custom name for the synchronization task.
Synchronization Method: Select Single-Table Real-Time.
Synchronization Mode: Select Full Sync.
Step 2: Configure network and resources
In the Network and Resource Configuration section, select the Resource Group that the synchronization task uses. You can allocate Task Resource Usage CUs to the task.
For Source Information, select an existing Hologres data source. For Destination, select an existing Hologres data source, and then click Test Connectivity.
After both the source and destination data sources pass the connectivity test, click Next.
Step 3: Configure the synchronization channel
Configure the source Hologres data source
At the top of the page, click the Hologres data source and edit Holo source information.
In the Holo source information section, select the schema that contains the Hologres table that you want to read, and then select the source table.
In the upper-right corner, click Data Sampling. In the Output Preview dialog box, specify Number of Sample Records and click Start Sampling to sample data from the Hologres source and preview the sampled data.
The sample data provides input for the data preview and visualization configuration of subsequent data processing nodes.
Preview the output of data processing nodes
Click the 
icon to add a data processing method. Five data processing methods are available: data masking, string replacement, data filtering, JSON parsing, and field editing and assignment. You can arrange these methods in any order. When the task runs, it processes data sequentially in the configured order.
After you configure a data processing node, click the Output Preview button in the upper-right corner. In the dialog box that appears, click Retrieve Upstream Output to simulate the result after the current data processing node processes the sampled Hologres data.
The data output preview depends on Data Sampling of the Hologres source. Complete data sampling in the Hologres source form before you run a data output preview.
Configure the destination Hologres data source
At the top of the page, click the Hologres data destination and edit Destination Information.
In the Destination Information section, select the schema that contains the Hologres table that you want to write to, and specify whether to automatically create the destination table (Create tables automatically) or to use an existing table (Use Existing Table).
If tables are created automatically, the destination table uses the same name as the source table by default. You can change the name of the destination table manually.
If you use an existing table, select the destination table that you want to synchronize from the drop-down list.
(Optional) Edit the table structure.
When you select Create tables automatically, click the Edit Table Schema button to edit the target table structure in the dialog box. You can also click Regenerate Schema from Upstream Node to automatically generate the table structure from the upstream node's output columns. In the generated structure, you can select a column to set as the primary key.
The target table must have a primary key. Otherwise, you cannot save the configuration.
Set Job Type and Write Conflict Policy.
Job Type:
Replay: Provides mirroring. When a record is inserted at the source, a record is also inserted in Hologres. When a record is updated or deleted at the source, the corresponding record is updated or deleted in Hologres.
Insert: Treats Hologres as streaming storage. All data from the source is saved by using INSERT operations.
Write Conflict Policy: The policy that is used when a data write conflict occurs. Valid values: Overwrite and Ignore.
Configure field mapping.
The system automatically generates mappings between upstream columns and target columns based on the The same name mapping principle. You can adjust these mappings as needed. A single upstream column can be mapped to multiple target columns, but multiple upstream columns cannot be mapped to a single target column. If an upstream column is not mapped to a target column, its data will not be written to the target table.
4. Alerts
To prevent task errors from causing delays in business data synchronization, you can set an alert policy for the synchronization task.
-
Click Alert Settings in the upper-right corner of the page to open the Alert Rule Configurations for Real-time Synchronization Subnode settings page.
-
Click Add Alert Rule to configure an alert rule.
NoteThe alert rules you define here apply to the real-time synchronization subtasks that this task generates. After you configure the task, you can view and modify the alert rules for these subtasks on the Run and manage real-time synchronization tasks page.
-
Manage alert rules.
For existing alert rules, you can use the toggle switch to enable or disable them. You can also send alerts to different recipients based on the alert level.
5. Advanced settings
The synchronization task provides several parameters that you can modify as needed.
Before making changes, ensure that you fully understand the function of each parameter to prevent unexpected errors or data quality issues.
-
Click advanced settings in the upper-right corner of the page to open the advanced settings page.
-
On the advanced settings page, modify the parameter values as needed.
6. Resource group
You can click Configure Resource Group in the upper-right corner to view and switch the task's current resource group.
7. Trial run
After you configure the task, click Dry Run in the upper-right corner. This feature simulates the entire task on a small data sample and allows you to preview the results in the target table. If there are configuration errors, runtime exceptions, or dirty data, you will receive real-time error messages. This helps you quickly verify that the task is configured correctly and produces the expected results.
-
In the dialog box, set the sampling parameters: Start Time and Number of Sample Records.
-
Click Start Sampling to collect the sample data.
-
Click Preview to simulate the entire task processing using the sampled data.
8. Run the synchronization task
-
After completing all settings, click Complete at the bottom of the page.
-
On the page, find the task you created and click Start in the Operations column.
-
Click the Name/ID of the corresponding task in the Task List to view its detailed execution process.
Start the synchronization task
On the synchronization task page, click Start in the Actions column to start the synchronization task. To manage the task and view its running status, see Manage and monitor synchronization tasks.
Task O&M
Manage and monitor synchronization tasks
On the synchronization task page, you can view the list of synchronization tasks and the basic information of each task.
In the Actions column, click Start or Stop to start or stop a synchronization task. Under More, operations such as Edit and View are available.
For a started task, you can view the basic running status in Execution Overview and click the corresponding overview area to view the execution details.
A single-table real-time synchronization task from Hologres to Hologres consists of three steps:
Schema Migration: Shows how the destination table is created, either as an existing table or by automatic table creation. If tables are created automatically, the table creation DDL is displayed.
Full Data Initialization: If you set Synchronization Mode to Full Sync, the progress of full initialization is displayed.
Real-time Data Synchronization: Shows the performance statistics of real-time synchronization, including real-time read/write traffic, dirty data, failover, and run logs.
Rerun a task
In special cases, such as when you need to modify synchronized fields or adjust target table information, you can click Rerun in the Operations column of the synchronization task. This action synchronizes the adjusted fields and other changes to the target. The process skips unchanged, previously synchronized tables.
-
To run the task again without any changes, click Rerun.
-
If you edit the task, click Complete after making your changes. The task's action changes to Apply Updates. Clicking Apply Updates reruns the task with the new configuration.