DataWorks:Single-table real-time synchronization

Capability

Description

Data synchronization across various data sources

Real-time synchronization supports a wide range of data sources. You can combine various source and destination data sources to build a synchronization pipeline. For more information, see Supported data sources and sync solutions.

Data synchronization in complex network environments

Real-time synchronization supports various environments, including Alibaba Cloud databases, on-premises IDCs, self-managed databases on ECS, and databases from other cloud providers. Before you start, ensure Network Connectivity between the Resource Group and the source and destination endpoints. For configuration details, see Network connectivity solutions.

Use cases

Real-time synchronization supports both single-table-to-single-table synchronization and synchronizing incremental data from sharded sources to a single destination table.

• Data Integration & Data Studio (new version): Provides wizard-based configuration for single-table ETL synchronization. This method offers rich data processing capabilities and advanced features like data sampling, Simulation Run, and advanced parameter settings.

• Data Studio (legacy): Provides a drag-and-drop interface to configure single-table ETL synchronization. This option supports data processing features like Data Filtering, String Replacement, and Data Masking.

Real-time synchronization task configuration

When configuring a real-time synchronization task, you can use the following capabilities to perform codeless, real-time ETL on single-table data. For more information, see Configure a real-time sync task in Data Integration.

Real-time Synchronization for Single Tables:

• Configuration methods: Offers a drag-and-drop graphical interface or a wizard for low-code development. The codeless approach makes it easy for new users to get started.

• Field Mapping: Supports Same-name Mapping, positional mapping, and custom field relationships. For source fields without a corresponding destination field, you can configure a policy to add a new column, ignore the field, or report an error. You can also dynamically assign values to destination fields by using Constants, Variables, and Functions.

• Data processing: Supports features for processing source data, such as Data Filtering, String Replacement, Data Masking, and JSON Parsing, before the data is written to the destination database.

• Runtime Debugging: Supports data sampling from the source and provides intermediate results at each processing step. You can use the Simulation Run feature to preview the final data output. Data generated during a Simulation Run is not written to the destination table, preventing any impact on your production data.

Task O&M for real-time synchronization

You can configure monitoring and alerting for synchronization tasks.

• Supports Breakpoint Resumption. If a task is interrupted by an exception, you can resume from a specified checkpoint to ensure data integrity.

• You can configure monitoring and alerting for business latency, failover, DDL policies, and Heartbeat Detection. For more information, see Real-time sync task O&M.

• Alarm notifications are also sent to alarm recipients via email, SMS, phone calls, and DingTalk, allowing you to promptly detect and resolve task anomalies.

• Supports alert fatigue control. You can configure a rule to limit notifications to one per specified interval, preventing an excessive number of alerts.

• Supports Heartbeat Detection. The heartbeat alert feature is automatically enabled or disabled when the task starts or stops. Manual changes to this setting are retained.