DataWorks:Single-table real-time synchronization capabilities

Capability

Description

Data synchronization across various data sources

Real-time synchronization supports various data sources. You can combine different source and destination data sources to create a synchronization pipeline. For more information, see Supported data sources and sync solutions.

Data synchronization in complex network environments

Real-time synchronization supports data transfer from Alibaba Cloud databases, on-premises data centers, self-managed databases on ECS instances, and databases outside Alibaba Cloud. Ensure that network connectivity exists between your resource group and both the source and destination before you start. For configuration details, see Network connectivity solutions.

Synchronization scenarios

Real-time synchronization supports syncing a single table to a destination table and consolidating incremental data from sharded databases and tables into one destination table.

• Data Integration & Data Studio (new version): Wizard-based configuration for single-table-to-single-table ETL synchronization. In addition to rich data processing features, it also supports advanced features such as data sampling, dry run, and advanced parameters.

• Data Studio (legacy): Drag-and-drop configuration for single-table-to-single-table ETL synchronization. It supports data processing features such as data filtering, string replacement, and data masking.

Real-time synchronization task configuration

The following capabilities are supported when you configure a real-time synchronization task. You can collect single-table ETL real-time data through simple task configuration without writing code. For more information, see Configure a single-table real-time synchronization task.

Single-table real-time synchronization:

• Configuration method: Supports graphical drag-and-drop or wizard-based low-code development. No coding is required, making it easy for beginners to get started.

• Field mapping: Supports same-name mapping and same-row mapping, as well as custom field relationships. If a source field has no corresponding field in the destination table, you can specify a dynamic field handling policy to add a column, ignore, or report an error. The synchronization task also provides dynamic assignment of constants, variables, and functions to destination fields.

• Data processing: Supports processing source data with Data Filtering, Replace String, Data Masking, and JSON Parsing, and then outputs the processed data to the destination database.

• Code debugging: Supports sampling data from the source data source and outputting intermediate results at each data processing step. You can use Dry Run to simulate the final data output. Data output during Dry Run is not written to the destination table, so you do not need to worry about the impact of debugging on actual data.

Real-time synchronization task O&M

Supports configuring monitoring alerts for synchronization tasks.

• Supports checkpoint-based resumption. When a task is interrupted or data is lost due to unexpected fluctuations, you can specify an appropriate checkpoint to ensure data integrity.

• Supports configuring monitoring alerts for business latency, failover, DDL policies, and heartbeat checks. For more information, see Configure monitoring alerts for a real-time synchronization task.

• Alert notifications are sent to alert recipients through emails, text messages, phone calls, and DingTalk messages, allowing you to detect and handle task exceptions in a timely manner.

• Supports alert fatigue control. To prevent a large number of alerts from being generated within a short period of time, DataWorks allows you to configure a rule to send only one alert within a specified time interval.

• Supports heartbeat detection, which automatically enables or disables heartbeat alerts when the task starts or stops. If you manually disable the feature, it remains disabled.

DI Agent

Use DI Agent to create and manage single-table real-time synchronization tasks through conversations. Instead of filling out configuration forms field by field, you can describe your synchronization requirements in natural language, and the Agent automatically handles data source identification, field mapping, resource group assignment, and scheduling policy configuration. It also provides O&M capabilities such as intelligent diagnostics and periodic inspections. For more information about DI Agent, see DI Agent overview.