DataWorks:Configure a real-time full-database synchronization task

Actions

Description

Refresh

The system automatically lists the source tables you selected, but the specific attributes of the destination tables take effect only after you refresh and confirm the mapping.

• Select the tables to be synchronized in batches and click Bulk Refresh Mapping.

• Destination Table Name: The destination table name is automatically generated based on the Customize Mapping Rules for Destination Table Names rule. The default is ${source_database_name}_${table_name}. If a table with this name does not exist in the destination, the system automatically creates one.

Customize Mapping Rules for Destination Table Names (optional)

The system has a default table name generation rule: ${source_database_name}_${table_name}. You can also click the Edit button in the Customize Mapping Rules for Destination Table Names column to add a custom rule for destination table names.

• Rule name: Define a name for the rule. We recommend using a descriptive name.

• Destination table name: You can generate the destination table name by clicking the icon and selecting Manually enter or Built-in Variable to concatenate parts of the name. Variables include the source data source name, source database name, and source table name.

• Edit built-in variable: You can perform string conversions on existing built-in variables.

This feature supports the following scenarios:

1. Add prefixes or suffixes: Add a prefix or suffix to the source table name by setting a constant.

   Rule configuration	Result

2. Unified string replacement: Replace the string dev_ in the source table name with prd_.

   Rule configuration	Result

3. Write data from multiple tables to one table: Set the destination table name as a constant.

   Rule configuration	Result

Edit field type mapping (optional)

The system provides default mappings between source and destination field types. You can click Edit Mapping of Field Data Types in the upper-right corner of the table to customize the mapping relationship. After configuration, click Apply and Refresh Mapping.

When editing the field type mapping, ensure that the type conversion rules are correct. Otherwise, type conversion may fail, leading to dirty data and task interruptions.

Edit destination table structure (optional)

Based on the custom table name mapping rules, the system creates new destination tables or reuses existing ones with the same name.

DataWorks generates the destination schema based on the source schema. Manual intervention is usually not required. You can also modify the table schema in the following ways:

• Add a field to a single table: Click the button in the Target Table column to add a field.

• Add fields in batches: Select all tables to be synchronized and choose Batch Edit > Destination Table Schema - Batch Modify and Add Field at the bottom of the table.

• Renaming columns is not supported.

For existing tables, you can only add fields. For new tables, you can add fields, partition fields, and set table types or properties. For more details, refer to the editable areas in the UI.

Value assignment

Native fields are automatically mapped based on fields with the same name in the source and destination tables. You must manually assign values for newly added fields and partition fields from the previous steps. Perform the following operations:

• Assign values to a single table: Click the Configuration button in the Value assignment column to assign values to the destination table fields.

• Assign values in batches: At the bottom of the list, choose Batch Edit > Value assignment to assign values to the same fields across multiple destination tables in batches.

You can assign constants and variables by switching the type in Value Type. The following methods are supported:

• Table field

  • Manual Input: Enter a constant value directly, such as abc.

  • Select variable: Select a supported system variable from the drop-down list. You can view the specific meaning of each variable in the tooltip in the UI.

  • Function: You can use functions to perform simple transformations on the destination field. For more information, see Use function expressions to assign values to destination table fields.

• Partition field: You can dynamically create partitions based on the enumerated values of a source field or the event time.

  • Manual input: Enter a constant value directly, such as abc.

  • Source field: Use the value of a source table field as the partition field value. The value type can be a field value or a time value.

    • Field value: The enumerated values from a source field. We recommend using a field with a limited number of distinct values to prevent creating too many partitions and dispersing the data excessively.

    • Time value: If the source field contains time values, you can process them based on different formats and specify the Target format to format the partition value.

      • Time string: A string that represents time, such as "2018-10-23 02:13:56" or "2021/05/18". You can serialize it into a time format by specifying the source and destination time formats. For the preceding examples, you can use formats like yyyy-MM-dd HH:mm:ss and yyyy/MM/dd to perform the serialization.

      • Time object: If the source value is already in a time-type format such as Date or Datetime, select this type directly.

      • Unix timestamp (seconds): A second-level timestamp. Numbers or strings that conform to the 10-digit timestamp format are also supported, such as 1610529203 and "1610529203".

      • Unix timestamp (milliseconds): A millisecond-level timestamp. Numbers or strings that conform to the 13-digit timestamp format are also supported, such as 1610529203002 and "1610529203002".

  • Select variable: You can use the source event change time, EVENT_TIME, as the source for the partition value. This is used in a similar way to the Source Field option.

  • Function: You can use functions to perform simple transformations on the source field and use the result as the partition value. For more information, see Use function expressions to assign values to destination table fields.

Source Split Column

You can select a field from the source table in the Source Split Column drop-down list, or select Not Split. During execution, the synchronization task is divided into multiple subtasks based on this field to enable concurrent, batched data reading.

We recommend using the table's primary key as the source split column. String, float, and date types are not supported.

Currently, source split columns are supported only for MySQL sources.

Skip full synchronization for table

If you have configured full synchronization in the task steps, you can choose to skip it for a specific table. This is useful when the full data has already been synchronized to the destination through other means.

Full condition

This allows you to filter the source data during the full synchronization phase. You only need to write the WHERE clause content, not the WHERE keyword itself.

Configure DML Rule

DML message processing allows you to filter and control the captured data changes (Insert, Update, Delete) from the source before they are written to the destination. This rule is effective only during the incremental synchronization phase.

Other

Table Type: MaxCompute supports Standard Table, PK Delta Table, and Append Delta Table. If the destination table has not yet been created, you can select the table type when you edit the table structure. The type of an existing table cannot be changed.

• The real-time full + incremental mode for an entire database only supports destination tables of the PK Delta Table type.

• Replay mode in full incremental mode supports PK Delta Table. Incremental stream mode supports Standard Tables and Append Delta Table.

For a detailed introduction to Delta Tables, see Delta Table.