DataWorks:Configure a real-time full database synchronization task

Operation

Description

Refresh

The system automatically lists the source tables you selected. However, the specific properties of the target tables take effect only after you refresh and confirm them.

• Select the tables to sync in batches and click Batch Refresh Mapping.

• Target Table Name: The target table name is automatically generated based on the Customize Mapping Rules rule. The default is ${Source_Database_Name}_${Table_Name}. If a table with this name does not exist in the target, the system automatically creates it for you.

Customize Target Table Name Mapping (Optional)

The system has a default table name generation rule: ${Source_Database_Name}_${Table_Name}. You can also click the Edit button in the Customize Mapping Rules column to add a custom rule for target table names.

• Rule Name: Define a name for the rule. We recommend giving the rule a name with clear business meaning.

• Target Table Name: You can create the target table name by clicking the button and combining a Manual Input and a Built-in Variable. Supported variables include the source data source name, source database name, and source table name.

• Edit Built-in Variable: You can perform string transformations on the original built-in variables.

This allows for the following scenarios:

1. Add a prefix or suffix: Add a prefix or suffix to the source table name by setting a constant.

   Rule configuration	Result

2. Uniform string replacement: Replace the string "dev_" in the source table name with "prd_".

   Rule configuration	Result

3. Write multiple tables to a single table.

   Rule configuration	Result

Edit Mapping of Field Data Types (Optional)

The system has a default mapping between source and target field types. You can click Edit Mapping of Field Data Types in the upper-right corner of the table to customize the mapping relationship. After configuration, click Apply And Refresh Mapping.

When editing field type mappings, ensure the conversion rules are correct. Otherwise, type conversion failures can produce dirty data and affect task execution.

Edit Target Table Schema (Optional)

The system automatically creates a target table if it does not exist, or reuses an existing table with the same name, based on the custom table name mapping rule.

DataWorks automatically generates the target table schema based on the source table schema. In most scenarios, no manual intervention is needed. You can also modify the table schema in the following ways:

• To add a field to a single table, click the button in the Destination Table Name column.

• Add fields in batches: Select all tables to be synced, and in the menu at the bottom of the table, choose Batch Modify > Destination Table Schema - Batch Modify and Add Field.

• Renaming columns is not supported.

For existing tables, you can only add fields. For new tables, you can add fields and partition fields, and set the table type or table properties. For more information, see the editable areas in the interface.

Assign Values To Target Table Fields

Native fields are automatically mapped based on matching field names in the source and target tables. You must manually assign values for the new fields and partition fields added in the previous step. Perform the following operations:

• Assign values to a single table: Click the Configure button in the Value assignment column to assign values to the target table fields.

• Assign values in batches: In the menu at the bottom of the list, choose Batch Modify > Value assignment to assign values to the same fields in multiple target tables in batches.

You can assign constants and variables. Switch the type in Manually Assign Value. The following methods are supported:

• Field in Destination Table

  • Manual assignment: Directly enter a constant value, such as abc.

  • Select variable: Select a system-supported variable from the drop-down list. You can view the meaning of each variable in the tooltip on the interface.

  • Function: Use functions for simple transformations on the target field. For more information about how to use functions, see Use function expressions to assign values to target table fields.

• Partition Fields: You can use the enumerated values of a source field or the event time as the partition value to dynamically create partitions.

  • Manually Assign Value: Directly enter a constant value, such as abc.

  • Source Field: Use the value of a source table field as the partition field value. The value type can be a field value or a time value.

    • Field Value: The enumerated values of the source field. We recommend using fields with a limited number of enumerated values to prevent creating too many partitions and dispersing data too widely.

    • Time Value: If the value in the source field is a time, you can process it according to different formats and specify a Destination Forma to format the partition value.

      • Time String: A string that represents a time, such as "2018-10-23 02:13:56" or "2021/05/18". Serialize it into a time by specifying the source and target time formats. For the examples above, you can use the formats yyyy-MM-dd HH:mm:ss and yyyy/MM/dd for serialization and recognition.

      • Time Object: If the source value is already in a time format such as Date or Datetime, select this type directly.

      • Unix Timestamp (seconds): A timestamp in seconds. It also supports numbers or strings that match the 10-digit timestamp format, such as 1610529203 or "1610529203".

      • Unix Timestamp (milliseconds): A timestamp in milliseconds. It also supports numbers or strings that match the 13-digit timestamp format, such as 1610529203002 or "1610529203002".

  • Select Variable: You can use the source event change time, EVENT_TIME, as the source for the partition value. The usage is similar to that of a source field.

  • Function: Use functions to perform simple transformations on the source field and use the result as the partition value. For more information about how to use functions, see Use function expressions to assign values to target table fields.

Source Split Column

You can select a field from the source table in the source sharding column drop-down list or select Not Split. When the sync task runs, it will be split into multiple tasks based on this field to read data concurrently and in batches.

We recommend using the table's primary key as the source sharding column. String, float, date, and other types are not supported.

Currently, the source sharding column is supported only when the source is MySQL.

Do Execute Once Offline

If you have configured full synchronization in step 3, you can choose to cancel the full data synchronization for a specific table. This is useful in scenarios where the full data has already been synced to the target using other methods.

Full Condition

Filter the source data during the full synchronization phase. Only write the where clause here. Do not include the where keyword.

Configure DML Rule

DML message processing is used for fine-grained filtering and control of change data (Insert, Update, Delete) captured from the source before it is written to the target. This rule only takes effect during the incremental phase.

Other

• Table Type: MaxCompute supports standard tables and Delta Tables. If the target table status is 'To be created', you can select the table type when editing the target table schema. The type of an existing table cannot be changed.

  Real-time full database synchronization only supports Delta Table as the target table type. For standard table types, see Full and incremental (Near Real-Time) synchronization tasks.

• If the table type is Delta Table, you can define the Table Bucket Num and Acid Data Retain Hours.

For a detailed introduction to Delta Tables, see Delta tables.