DataWorks:Batch synchronization of a MySQL database to Hive

Actions

Description

Refresh mapping

The system automatically lists your selected source tables, but destination table properties take effect only after you refresh and confirm.

• Select the tables to be synchronized in bulk, and click Bulk Refresh Mapping.

• Destination table name: The destination table name is automatically generated based on the Customize Mapping Rules for Destination Table Names rule. The default is ${SourceDatabaseName}_${TableName}. If no table with this name exists in the destination, DataWorks creates one for you.

Edit column type mapping (Optional)

The system provides default mappings between source and destination column types. You can click Edit Mapping of Field Data Types in the upper-right corner of the table to customize the mappings. After configuration, click Apply and Refresh Mapping.

When editing column type mapping, ensure the type conversion rules are correct. Otherwise, type conversion failures may generate Dirty Data and affect task execution.

Customize destination table name mapping (Optional)

The default table name generation rule is: ${SourceDatabaseName}_${TableName}. You can also click the Edit button in the Customize Mapping Rules for Destination Table Names column to add a custom rule.

• Rule Name: Define a name for the rule. Use a name with clear business meaning.

• Destination Table Name: You can generate the destination table name by clicking the icon and combining Manually enter and Built-in Variable. Supported variables include source data source name, source database name, and source table name.

• Edit Built-in Variables: You can perform string transformations on the original built-in variables.

This feature can be used for the following scenarios:

1. Add prefixes and suffixes to names: Set a constant to add a prefix or suffix to the source table name.

   Rule configuration	Result

2. Uniformly replace strings: Replace the string dev_ in the source table name with prd_.

   Rule configuration	Result

3. Write data from multiple tables to a single table.

   Rule configuration	Result

Customize destination database name mapping (Optional)

Some destination Data Sources, such as Hologres, support defining mapping rules for the destination database. The configuration method is similar to Customize destination table name mapping.

Customize destination schema name mapping (Optional)

Some destination Data Sources, such as Hologres, support defining mapping rules for the destination schema. The configuration method is similar to Customize destination table name mapping.

Edit destination Table Schema (Optional)

The system automatically generates the destination table schema based on the source table structure. In most cases, manual changes are unnecessary. If special handling is required, you can customize it as follows:

• To add a column to a single table, click the icon in the Target Table column.

• Add columns in bulk: Select all tables to be synchronized, and at the bottom of the table, select Batch Edit > Batch Modify Table Schema.

• Renaming columns is not supported.

Assign values to destination table Columns

Regular columns are automatically mapped based on matching names between the source and destination tables. You must manually assign values for Partition Columns and any newly added columns from the previous step. To do this:

• Assign for a single table: Click the Configuration button in the Value assignment column to assign values to the destination table columns.

• Assign in bulk: At the bottom of the list, select Batch Edit > Value assignment to assign values to the same columns in multiple destination tables.

You can assign both constants and variables. You can switch the assignment type for Table Columns and Partition Columns in the Value Type field. The supported methods are as follows:

• Manual assignment: Enter a constant value or a scheduling parameter directly, such as abc, or use the ${xxx} format for scheduling parameters, such as ${bizdate}.

• Select variable: Select a system-supported variable from the drop-down list. You can view the description of each variable in the tooltip on the interface.

Both variables and periodic scheduling parameters are automatically replaced with the corresponding date when the scheduler runs the task.

Set Source Chunking Column

In the Source Chunking Column dropdown, you can select a column from the source table or choose Disable. DataWorks splits the synchronization task into multiple sub-tasks based on this column for concurrent, batched data reading.

Use the table's primary key as the Source Chunking Column. String, float, date, and other types are not supported.

Currently, the Source Chunking Column is only supported when the source is MySQL.

Customize advanced parameters

You can set Runtime Configurations for each sub-task individually. Modify these parameters only if you fully understand their meaning to avoid unexpected issues such as task delays, excessive resource consumption blocking other tasks, or data loss.