DataWorks:Configure a full and incremental full-database synchronization task

Action

Description

Refresh Mapping

The system automatically lists the source tables you selected. You must then refresh the mapping to apply settings and view the destination table properties.

• Select the tables to synchronize in bulk and click Batch Refresh Mappings.

• Destination Table Name: The destination table name is automatically generated based on the Customize Destination Table Name Mapping rule. The default format is ${SourceDBName}_${TableName}. If a table with this name does not exist at the destination, the system creates it automatically.

Customize Destination Table Name Mapping (Optional)

The system has a default table naming rule: ${SourceDBName}_${TableName}. You can also click the Edit button in the Customize Destination Table Name Mapping column to add custom naming rules.

• Rule Name: Define a name for the rule. Use a name with a clear business meaning.

• Destination Table Name: You can construct the destination table name by clicking the icon and combining values from Manual Input and Built-in Variable. Supported variables include the source data source name, source database name, and source table name.

• Edit Built-in Variables: You can apply string transformations to the built-in variables.

You can achieve the following:

1. Add prefixes or suffixes: Set a constant to add a prefix or suffix to the source table name.

   Rule configuration	Application effect

2. Perform a bulk string replacement: For example, replace the string dev_ with prd_ in all source table names.

   Rule configuration	Application effect

3. Write data from multiple tables into a single table.

   Rule configuration	Application effect

Edit Field Type Mapping (Optional)

The system provides a default mapping between source and destination field types. You can click Edit Field Type Mapping in the upper-right corner of the table to customize the mapping. When you are finished, click Apply and Refresh Mapping.

When you edit the field type mapping, ensure that the type conversion rules are valid. Incorrect rules can cause type conversion failures, generate dirty data, and disrupt the task.

Edit Destination Table Schema (Optional)

The system automatically creates destination tables that do not exist or reuses existing tables with the same name, based on your custom naming rules.

DataWorks automatically generates the destination table schema based on the source table schema. In most cases, no manual changes are needed. However, you can modify the table schema in the following ways:

• Add a field to a single table: Click the icon in the Destination Table Name column to add a field.

• Add fields in bulk: Select all tables to be synchronized and choose Batch Modify > Destination Table Schema - Batch modify and add fields at the bottom of the list.

• Renaming columns is not supported.

For existing tables, you can only add fields. For new tables, you can add fields, partition fields, and set the table type or properties. For details, refer to the editable areas in the UI.

Assign a value to a destination table field

The system automatically maps fields that exist in both the source and destination tables by name. You must manually assign values for any newly added fields.

• Assign a value for a single table: Click the Configure button in the Assign a value to a destination table field column to assign a value to a destination field.

• Assign values in bulk: At the bottom of the list, choose Batch Modify > Assign a value to a destination table field to assign values to fields with the same name across multiple destination tables.

You can assign constants or variables. Switch the type in the Assignment Method menu. The following methods are supported:

• Table Field

  • Manual Assignment: Directly enter a constant value or a scheduling parameter, such as abc or ${bizdate}.

  • Select Variable: Select a system-provided variable as the field value. For available variable types, see the descriptions in the UI.

  • Source Field: Use the value from a source table field. You can choose to use either the field value or a time value.

    • Field value: If the source field is not a time-based type, you can directly write its value to the destination field.

    • Time value: If the source field contains a time value, you can process it based on its format and specify a Target Format to format the output.

      • Time string: A string that represents a time, such as 2018-10-23 02:13:56 or 2021/05/18. You can parse it by specifying the source and destination time formats. For the examples above, you could use formats like yyyy-MM-dd HH:mm:ss and yyyy/MM/dd.

      • Time object: If the source value is already a time-based type, such as Date or Datetime, select this option.

      • Unix timestamp (seconds): A timestamp in seconds. Numbers or strings with a 10-digit timestamp format are also supported, such as 1610529203 and "1610529203".

      • Unix timestamp (milliseconds): A timestamp in milliseconds. Numbers or strings with a 13-digit timestamp format are also supported, such as 1610529203002 and "1610529203002".

  • Function: You can use functions to perform simple transformations on a source field before assigning it as the partition value. For more information about how to use functions, see Functions.

• Partition Field: You can dynamically create partitions based on the enumerated values of a source field or the event time.

  • Manual Assignment: You can enter a constant value or a scheduling parameter that you have configured in the Periodic Scheduling Parameters section, such as abc or ${bizdate}.

Set Source Shard Key

From the source shard key drop-down list, you can select a field from the source table or choose Do not shard. During synchronization, the task splits into multiple subtasks based on this key, which enables data to be read in parallel batches.

Use a field with an even data distribution, such as the table's primary key, as the shard key. String, float, and date types are not supported.

Currently, the source shard key is supported only when the source is MySQL.

Skip Full Synchronization

If you have already configured full synchronization, you can choose to skip the full synchronization for a specific table. This is useful if you have already synchronized the full data to the destination by other means.

Full Synchronization Condition

Applies a filter to the source data during the full load phase. Enter only the `WHERE` clause, without the `WHERE` keyword.

DML Rule Configuration

DML message handling allows you to finely filter and control the change data (Insert, Update, Delete) captured from the source before it is written to the destination. This rule applies only during the incremental phase.

Full Merge Period

Currently, only daily merges are supported. You can configure the specific time for the merge task schedule in Custom Merge Time.

Merge Primary Key

You can define the primary key by selecting one or more columns from the table.

• Single-column primary key: Select a single column with unique values (such as id) as the primary key.

• Composite primary key: If no single column is unique, you can select multiple columns. The combination of values in these columns must uniquely identify each row to ensure data uniqueness and prevent overwrites during writes.