DataWorks:Offline sync capabilities

Feature

Description

Full or incremental data synchronization

Configure Data Filtering and use scheduling parameters in offline sync tasks to perform full or incremental data synchronization. The configuration for incremental sync varies by plugin. For more information about configuring incremental data synchronization, see Scenario: Configure an incremental offline sync task.

Field mapping

Establish mapping rules between fields to write source data to the corresponding destination fields. Ensure that the data types of the source and destination fields are compatible during configuration.

• Multiple field mapping methods are available:

  • The codeless UI supports mapping by name and by row, and also allows custom field relationships. Data in unmapped fields is automatically ignored. Ensure that the corresponding destination fields have default values or allow null values to avoid write failures.

  • The code editor maps fields strictly based on the order in the column configuration. The number of fields in the reader and writer must be identical. Otherwise, the task will fail to execute.

• Sync tasks also provide a dynamic value assignment feature for destination fields. This supports flexible configuration of constants, scheduling parameters, and built-in variables, such as ${bizdate}. These parameters must be assigned their final values during the scheduling phase.

Job rate limit control

• The task concurrency control feature limits the maximum number of concurrent connections for reading from and writing to the database.

• The sync rate feature controls traffic to prevent excessive pressure on the source or destination data source. If no limit is set, the task uses the maximum transfer performance available in the current hardware environment.

Distributed task execution

Data sources that support distributed execution can use task segmentation technology to distribute a sync task across multiple nodes for concurrent execution. This allows the sync speed to scale linearly with the cluster size, breaking through single-node performance bottlenecks. This pattern is especially useful for high-throughput, low-latency sync scenarios. It also efficiently schedules idle cluster resources, significantly improving hardware utilization.

Dirty data policy

Dirty data refers to records that fail to be written to the destination due to exceptions, such as type conflicts or constraint violations. Offline sync supports defining a dirty data policy. You can define the number of tolerable dirty data records and their impact on the task.

• Ignore dirty data: Automatically filters out dirty data and writes only compliant data. The task continues to run.

• Tolerate a limited number of dirty data records: Set a threshold N. If the number of dirty data records is less than or equal to N, the abnormal records are discarded and the task continues. If the number exceeds N, the task fails and exits.

• Do not tolerate dirty data: The task fails and exits immediately if any dirty data is encountered.

Time zone

If the source and destination are in different time zones, set the source time zone to perform time zone conversion during synchronization.

Intelligent data processing

DataWorks supports data processing capabilities during data synchronization. This lets you transform and process source data before writing it to the destination:

String replacement: The offline sync task in DataWorks has a built-in string replacement feature. This lets you perform lightweight data transformations during data transfer without landing the data or requiring extra extract, transform, and load (ETL) steps.

AI-assisted processing: During data synchronization, you can integrate large AI models to perform semantic, sentiment, and other analyses on natural language from the source. The processed results are then written directly to the destination table.

Data vectorization: Extracts source data, creates vector embeddings, and writes them to a vector database.