This topic describes how to configure a Check node for a real-time synchronization task to trigger the running of the tasks that depend on the real-time synchronization task.
Limits
A real-time synchronization task that is created on the Synchronization Task page in Data Integration or on the DataStudio page is supported.
The following real-time synchronization channels are supported:
From MySQL to MaxCompute
From Kafka to MaxCompute
From MySQL to Hologres
Serverless resource groups are supported.
If the following error message appears when you use an old-version resource group, we recommend that you change the old-version resource group to a serverless resource group. You can also submit a ticket to contact technical support to upgrade the underlying version of your old-version resource group.
java.lang.RuntimeException: unknown type : 241 at com.alibaba.dataworks.messagewrapper.typehandler.TypeHandlerFactory.hander(TypeHandlerFactory.java:50) at com.alibaba.dataworks.messagewrapper.Main.main(Main.java:57) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.taobao.pandora.boot.loader.MainMethodRunner.run(MainMethodRunner.java:54) at com.taobao.pandora.boot.loader.Launcher.launch(Launcher.java:85) at com.taobao.pandora.boot.loader.Launcher.launch(Launcher.java:48) at com.taobao.pandora.boot.loader.SarLauncher.main(SarLauncher.java:167)
Node configuration
Description
The underlying version of your resource group may not meet requirements. Therefore, you must test whether a real-time synchronization task can be run as expected before you run a Check node.
The scheduling time of a Check node is used as the time when a check starts. The time when a generated instance is rerun depends on the time when the instance starts to be scheduled.
If a check performed by using the Check node is successful, the event time at which the data synchronization task is run to write data from the source to destination tables is later than the scheduling time for the node. In this case, you can write SQL statements for the destination tables.
Create a real-time synchronization task
Create a real-time synchronization task on the Synchronization Task page in Data Integration
Go to the Data Integration page.
Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose . On the page that appears, select the desired workspace from the drop-down list and click Go to Data Integration.
In the left-side navigation pane of the Data Integration page, click Synchronization Task. On the Synchronization Task page, select a source type from the Source drop-down list and a destination type from the Destination drop-down list, and click Create Synchronization Task.
NoteFor information about supported real-time synchronization channels, see the Limits section in this topic.
Configure the Synchronization Method parameter. For example, if you create a real-time synchronization task to synchronize data from MySQL to MaxCompute, you can set the Synchronization Method parameter to Real-time migration of entire database.
Configure other parameters that are required for the data synchronization task as prompted. For more information, see Common business scenarios and best practices.
Create a real-time synchronization task in DataStudio
Go to the DataStudio page.
Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose . On the page that appears, select the desired workspace from the drop-down list and click Go to Data Development.
In the Scheduled Workflow pane of the DataStudio page, find the desired workflow, right-click the workflow name, and then choose Create Node > Data Integration > Real-time Synchronization. In the Create Node dialog box, configure the Path and Name parameters and click Confirm.
In the Input and Output sections of the configuration tab of the real-time synchronization task, drag desired source and destination types to the canvas on the right. Then, separately click the source type and the destination type and configure the parameters in the panel that appears. For more information, see Create a real-time synchronization task in DataStudio.
Create a Check node
Go to the DataStudio page.
Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose . On the page that appears, select the desired workspace from the drop-down list and click Go to Data Development.
In the Scheduled Workflow pane of the DataStudio page, find the desired workflow, right-click the workflow name, and then choose Create Node > General > Check Node. In the Create Node dialog box, configure the Path and Name parameters and click Confirm.
On the configuration tab of the Check node, configure parameters.
Check Object: Select Real-time Synchronization Task.
Real-time Synchronization Task: Select the task that was created in the Create a real-time synchronization task section in this topic.
Policy for Stopping Check: The policy for stopping a check task on the current Check node. If the check is still not passed after the stopping condition that is specified in the check policy is met, the check task exits and enters the failed state. For more information, see the Step 2: Configure a check policy for the Check node section of the "Configure a Check node" topic.
Check results
Check passed
The check is performed at an interval of 5 minutes. The time at which a test is performed is
20241008143000(UTC+8 timestamp:1728369000000).
The check is passed when the latency of the real-time synchronization task is low. The timestamp is
1728369386000.
Check failed
The check is triggered twice when the latency of the real-time synchronization task is high. If the check is still not passed, the data synchronization task also fails.
