DataWorks:Configure a full and incremental full-database synchronization task

Actions

Description

Refresh

The system automatically lists the source tables you selected. However, you must refresh the mapping for the destination table properties to take effect.

• Select the tables to be synchronized in batches and click Refresh Mapping Results.

• Destination Table Name: The destination table name is automatically generated based on the rule in Customize Mapping Rules for Destination Table Names. The default format is ${SourceDatabaseName}_${TableName}. If a table with this name does not exist at the destination, the system automatically creates it.

Customize Mapping Rules for Destination Table Names (Optional)

The system has a default table naming rule: ${SourceDatabaseName}_${TableName}. You can click the Edit button in the Customize Mapping Rules for Destination Table Names column to add a custom rule.

• Rule Name: Define a name for the rule. Use a name that clearly indicates its business purpose.

• Destination Table Name: You can construct the destination table name by clicking the icon and combining options from Manual Input and Built-in Variable. The variables include the source data source name, source database name, and source table name.

• Edit Built-in Variable: You can apply string conversions on the built-in variables.

This feature supports the following scenarios:

1. Add a prefix or suffix to a name: Add a constant prefix or suffix to the source table name.

   Rule configuration	Result

2. Perform a uniform string replacement: Replace the string dev_ in all source table names with prd_.

   Rule configuration	Result

3. Write data from multiple tables to a single table.

   Rule configuration	Result

Edit Mapping of Field Data Types (Optional)

The system provides a default mapping between source and destination field data types. You can click Edit Mapping of Field Data Types in the upper-right corner of the table to customize the mapping. After configuration, click Apply and Refresh Mapping.

When you edit the data type mapping, ensure that the conversion rules are correct. An incorrect rule can cause type conversion failures, generate dirty data, and disrupt task execution.

Edit Destination Table Structure (Optional)

The system automatically creates a destination table if it does not exist, or uses an existing table with the same name, based on your custom table name mapping rules.

DataWorks automatically generates the destination table structure based on the source table structure. In most cases, no manual intervention is required. You can also modify the table structure in the following ways:

• Add a field to a single table: Click the icon in the Destination Table Name column to add a field.

• Add fields in batches: Select all the tables you want to synchronize, and then select Batch Modify > Destination Table Schema - Batch Modify and Add Field at the bottom of the list.

• You cannot rename columns.

For existing tables, you can only add fields. For new tables, you can add fields, partition fields, and set table types or properties. For more information, see the editable sections in the UI.

Value assignment

Native fields are automatically mapped based on matching field names between the source and destination tables. You must manually assign values to any new fields that you added in the previous step. To do this:

• Assign a value for a single table: Click the Configuration button in the Value assignment column to assign a value to a destination table field.

• Assign values in batches: At the bottom of the list, select Batch Modify > Value assignment to assign values to the same field across multiple destination tables.

You can assign constants or variables. Switch the type in the Value Type dropdown list. The following methods are supported:

• Table Field

  • Manual assignment: Enter a constant value or a scheduling parameter, such as abc or ${bizdate}.

  • Select variable: Select a system-provided variable as the field value. For more information about the variable types, see the descriptions in the UI.

  • Source Field: Use the value of a source table field. You can choose to use either the field value or a time value.

    • Field Value: If the source field is not a time-based type, you can write its value directly to the destination field.

    • Time Value: If the source field contains time-related values, you can process it based on its format and specify the Destination Format to format the output value.

      • Time String: A string that represents a time, such as 2018-10-23 02:13:56 or 2021/05/18. You can serialize these strings into time values by specifying the source and destination time formats. For the examples above, you can use formats like yyyy-MM-dd HH:mm:ss and yyyy/MM/dd.

      • Time Object: If the source value is already a time-based data type like Date or Datetime, select this type.

      • Unix Timestamp (seconds): A 10-digit, second-level timestamp, such as 1610529203 or "1610529203".

      • Unix Timestamp (milliseconds): A 13-digit, millisecond-level timestamp, such as 1610529203002 or "1610529203002".

  • Function: Use functions to perform simple transformations on source field values before assigning them to the destination field. For more information, see Use function expressions to assign values to destination table fields.

• Partition Field: You can dynamically create partitions based on enumerated values or the event time of a source field.

  • Manual assignment: You can enter a constant value or a scheduling parameter that you have already set in the Configure Scheduling Parameters for Periodical Scheduling configuration, such as abc or ${bizdate}.

Source Split PK

From the Source Split PK dropdown list, you can select a field from the source table or select Not Split. When the synchronization task runs, the task splits into multiple subtasks based on this field to read data concurrently.

Use a field with evenly distributed data, such as a primary key, as the split column. String, float, and date types are not supported.

Currently, the Source Split PK feature is supported only when the source is MySQL.

Execute Full Synchronization

If you have configured full data synchronization in step 3, you can clear this option to skip the full data synchronization for a specific table. This is useful if you have already synchronized the full dataset to the destination using another method.

Full condition

Apply a filter condition to the source during the full load phase. Enter only the WHERE clause content, without the WHERE keyword.

Configure DML Rule

Use DML message processing to filter and control the captured data changes (Insert, Update, and Delete) from the source before they are written to the destination. This rule applies only during the incremental synchronization phase.

Full Data Merge Cycle

Currently, only daily merging is supported. You can configure the specific scheduling time for the merge task in the Custom merge time setting.

Merge Primary Key

You can define a primary key by selecting one or more columns from the table.

• Single-column primary key: Select a single column with unique values, such as id, as the primary key.

• Composite primary key: If no single unique column exists, you can select multiple columns to form a composite key. The combination of values in these columns must uniquely identify each row to ensure data uniqueness and prevent data overwrites.