DataWorks:Configure whole database real-time synchronization tasks

Parameter

Description

Refresh Mapping Results

The system lists the selected source tables. However, destination attributes only take effect after you refresh and confirm the mapping.

• Select the tables to be synchronized in batches and click Batch Refresh Mapping.

• Destination table name: The target table name is automatically generated based on the Customize Mapping Rules for Destination Table Names rule. The default is ${Source DB Name}_${Table Name}. The system creates destination tables if they do not exist.

Customize Mapping Rules for Destination Table Names (Optional)

The system uses a default rule to generate table names: ${Source DB Name}_${Table Name}. You can also click Edit in the Customize Mapping Rules for Destination Table Names column to add custom target table name rules.

• Rule name: Define the rule name. Use a name with clear business meaning.

• Destination table name: Generate the target table name by clicking the button and selecting Manual Input or Built-in Variable for concatenation. Supported variables include the source data source name, source database name, and source table name.

• Edit built-in variable: You can perform string conversions on built-in variables.

Supported scenarios:

1. Add prefix/suffix: Add a prefix or suffix to the source table name by setting a constant.

   Rule configuration	Application effect

2. Unified string replacement: Replace the string "dev_" in the source table name with "prd_".

   Rule configuration	Application effect

3. Multiple tables to one table.

   Rule configuration	Application effect

Edit Mapping of Field Data Types (Optional)

The system provides default mappings between source and destination field types. You can click Edit Mapping of Field Data Types in the upper-right corner of the table to customize the mapping relationship, and then click Apply and Refresh Mapping.

Note: Incorrect field type conversion rules can cause conversion failures, data corruption, or task interruptions.

Edit Destination Table Structure (Optional)

The system automatically creates destination tables that do not exist or reuses existing tables with the same name based on the custom table name mapping rules.

DataWorks generates the destination schema based on the source schema. Manual intervention is usually unnecessary. You can also modify the table schema in the following ways:

• Add field to single table: Click the button in the Destination Table Name column to add a field.

• Batch add fields: Select all tables to be synchronized, and choose Batch Modify > Destination Table Structure - Batch Modify and Add Fields at the bottom of the table.

• Renaming columns is not supported.

For existing tables, you can only add fields. For new tables, you can add fields, partition fields, and set table types or properties. For more information, see the editable area in the interface.

Value assignment

Native fields are automatically mapped based on fields with the same name in the source and destination tables. You must manually assign values to Added Fields and Partition Fields. The procedure is as follows:

• Single table assignment: Click Configure in the Value assignment column to assign values to target table fields.

• Batch assignment: Select Batch Modify > Value assignment at the bottom of the list to assign values to the same fields in the destination tables in batches.

Assign constants or variables, and switch the type in Assignment Method. The supported methods are as follows:

• Table Field

  • Manual assignment: Directly input a constant value, such as abc.

  • Select variable: Select a supported system variable from the drop-down list. You can view the specific meaning of the variable in the tooltip in the interface.

  • Function: Supports simple transformation of the target field using functions. For usage details, see Assign values to target table fields using function expressions.

• Partition Field: Supports creating partitions dynamically based on the enumeration value of the Source Field or Event Time.

  • Manual Assignment: Directly input a constant value, such as abc.

  • Source Field: Uses the value of a source table field as the partition field value. The value type can be Field Value or Time Value.

    • Field Value: The enumeration value of the source field. We recommend that you use a field with a limited number of enumeration values to prevent creating too many partitions and dispersing data.

    • Time Value: If the value in the source field is time, you can process it according to different formats and specify the Destination Format to format the partition value.

      • Time String: A string representing time, such as "2018-10-23 02:13:56" or "2021/05/18". You can serialize it into a time format by specifying the source and destination patterns. For the example strings above, you can use the formats yyyy-MM-dd HH:mm:ss and yyyy/MM/dd for serialization and identification.

      • Time Object: If the source value is already in a time type format such as Date or Datetime, select this type directly.

      • Unix Timestamp (Seconds): A second-level timestamp. It also supports numbers or strings that conform to the 10-digit timestamp format, such as 1610529203 or "1610529203".

      • Unix Timestamp (Milliseconds): A millisecond-level timestamp. It also supports numbers or strings that conform to the 13-digit timestamp format, such as 1610529203002 or "1610529203002".

  • Select Variable: You can use the source event change time EVENT_TIME as the partition value source. The usage is similar to Source Field.

  • Function: Supports simple transformation of the source field using functions before using it as a partition value. For usage details, see Assign values to target table fields using function expressions.

Source Split Column

You can select a field from the source table in the drop-down list or select Not Split. When the synchronization task is executed, it is split into multiple tasks based on this field to allow concurrent and batched data reading.

We recommend that you use the table primary key as the source split key. Types such as string, float, and date are not supported.

Currently, source split keys are supported only when the source is MySQL.

Execute Full Synchronization

If full synchronization is configured in Step 3, you can individually cancel full data synchronization for a specific table. This applies to scenarios where full data has already been synchronized to the destination by other means.

Full Condition

Filter the source data during the full phase. You only need to write the where clause here, without the WHERE keyword.

Configure DML Rule

DML message processing allows for refined filtering and control of captured data changes (Insert, Update, Delete) before the data is written to the destination. This rule applies only during the incremental phase.

Others

• Table Type: MaxCompute supports Standard Table and Delta Table. If the target table status is "To Be Created", you can select the table type when editing the target table structure. Existing tables cannot change their type.

  Whole database real-time supports only Delta Table as the target table type. For Standard Table types, see Configure full and incremental synchronization.

• If the table type is Delta Table, you can define the Table Bucket Num and Historical Data Retention (Hours).

For more information about Delta Table, see Delta Table.