DataWorks:Configure a single-table batch synchronization task in wizard mode

1. Source

In the source section, configure the data table and fill in the required parameters as prompted.

2. Data processing

Data processing allows you to process data from the source table by using methods such as string replacement, AI-assisted processing, and data vectorization, and then write the processed data to the destination table.

Take string replacement as an example. The configuration items include Name and Description. In the replacement rules, select the Field Name, enter the Content to Replace (which supports Regular Expression Matching and Case-Sensitive Matching), and specify the Replacement Content. You can click Add Rule to add multiple replacement rules, and use Data Output Preview in the upper-right corner to view the processing results.

1. Click the toggle button to enable data processing.

2. In the Data Processing List, click Add Node and select a data processing type: Replace String, AI-Assisted Processing, or Data Vectorization. You can add multiple data processing nodes. DataWorks processes them in order.

3. Configure the data processing rules as prompted. For information about AI-assisted processing and data vectorization, see Data processing.

   Note

   Data processing requires additional compute resources, which increases the resource consumption overhead. It also increases the duration of the data synchronization task. Minimize the complexity of the processing to avoid affecting synchronization efficiency.

3. Destination

In the destination section, configure the data table and fill in the required parameters as prompted.

MaxCompute partition table configuration notes

When the destination is a MaxCompute partition table, note the following:

• Partition column identification: DataWorks automatically identifies the partition structure of the MaxCompute destination table. If only some partition columns are displayed in the UI, check whether all partition columns are correctly defined in both the development environment and the production environment. If the task fails and prompts you to configure table partition information, complete the partition parameters in the destination configuration.

• Column mapping refresh: If new columns are added to the source or destination but are not displayed in the column mapping section, try the following methods to refresh the cache:

  1. Make sure that the table schemas in both the development environment and the production environment are updated.

  2. On the configuration page, switch to a different table and then switch back to the original table to refresh the cache.

  3. If the cache is still not refreshed, restart the browser or use incognito mode to re-open the configuration page.

4. Configure column mappings

After you select the source and destination, you must specify the column mapping between the reader and writer. After you configure the column mappings, the task writes source columns to the corresponding destination columns based on the mappings.

• If a source column is not mapped to a destination column, the data in that source column is not synchronized to the destination.

• If the automatic mapping does not match your expectations, manually adjust the mappings.

• If you do not need a column mapping, you can manually delete the mapping line between the source and destination columns. The data in that source column will not be synchronized to the destination.

During synchronization, column type mismatches between the source and destination may generate dirty data, which prevents data from being written to the destination. For the number of dirty data records that can be tolerated during synchronization, configure the Advanced configuration in the next step.

Same-name mapping, same-row mapping, intelligent mapping, and rule-based mapping are supported. During configuration, you can also:

• Intelligent mapping: To improve configuration efficiency and reduce manual matching errors, Data Integration supports the intelligent column mapping feature. The system uses AI semantic analysis to automatically identify column names, data types, and comments of the source and destination tables, and recommends optimal mappings with one click. You only need to confirm the recommendations or make minor adjustments to complete the configuration.

  In the column mapping section, click Intelligent Mapping to open the intelligent mapping dialog. You can describe your mapping requirements in natural language.

  • Applicable scenario	Typical example	Recommended prompt
    Global semantic matching	Column names are completely different but have the same meaning (Example: user_id ↔ device_id)	Perform semantic matching on all columns of the source and destination tables, and automatically identify columns with the same meaning.
    Specific business domain matching	Only specific business columns need to be mapped (Example: only "user" or "order" related columns)	Map only the columns in the source table that contain "user information" (such as name, phone number, and ID) to the corresponding columns in the destination table. (Note: You can replace the keyword with "order", "logistics", "payment", etc.)
    Prefix/suffix convention differences	Core names are the same but prefixes/suffixes differ (Example: src_user_name ↔ tgt_user_name)	Ignore prefix and suffix differences in column names, and perform semantic matching based on core names only.
    Abbreviation and full name matching	One side uses abbreviations while the other uses full names (Example: amt ↔ amount)	Identify common English abbreviation-to-full-name mappings (such as amt=amount, addr=address) and create mappings accordingly.
    Exclude specific columns	Some columns are similar but do not need to be synchronized (Example: create_time is not needed)	Perform semantic matching, but exclude all columns that contain "time" or "log" in their names.
    Complex logic correction	The automatic matching result is incorrect and manual guidance is needed	Do not map the id column of the source table to the order_id column of the destination table. Regenerate the mapping suggestions.

  • After you enter the description, click Generate Preview. The system displays the suggested mappings in the Matching Result Preview section. You can review and select the mappings you need, and then click Apply to add the selected mappings to the column mapping. If you are not satisfied with the results, adjust the description and regenerate the preview.

• Rule-based mapping: When column names between the source and destination follow a pattern of differences, you can use the Rule-Based Mapping feature. By configuring rules such as prefix/suffix matching or character replacement, you can create column mappings in bulk. In the column mapping section, click Rule-Based Mapping, select the mapping method and rules, preview the mapping results after configuration, and then click Apply.

• Assign values to destination columns: You can add constants, scheduling parameters, and built-in variables to the destination table by clicking Add Fields in the Source Table Field column. For example, '123', '${scheduling parameter}', '#{built-in variable}#'.

  Note

  For more information about scheduling parameters, see Configure scheduling parameters.

• Add built-in variables: You can manually add built-in variables, map them to destination columns, and output the built-in variables to downstream tasks.

  The built-in variables available for each plug-in are as follows:

  Built-in variable	Description	Supported plug-in
  '#{DATASOURCE_NAME_SRC}#'	Source data source name	MySQL Reader MySQL (sharded) Reader PolarDB Reader PolarDB (sharded) Reader PostgreSQL Reader PolarDB-O Reader PolarDB-O (sharded) Reader
  '#{DB_NAME_SRC}#'	Name of the database where the source table resides	MySQL Reader MySQL (sharded) Reader PolarDB Reader PolarDB (sharded) Reader PostgreSQL Reader PolarDB-O Reader PolarDB-O (sharded) Reader
  '#{SCHEMA_NAME_SRC}#'	Name of the schema where the source table resides	PolarDB Reader PolarDB (sharded) Reader PostgreSQL Reader PolarDB-O Reader PolarDB-O (sharded) Reader
  '#{TABLE_NAME_SRC}#'	Source table name	MySQL Reader MySQL (sharded) Reader PolarDB Reader PolarDB (sharded) Reader PostgreSQL Reader PolarDB-O Reader PolarDB-O (sharded) Reader
  '#{FILE_NAME_SRC}#'	File name	OSS Reader HDFS Reader FTP Reader TOS Reader COS Reader S3 Reader Azure Blob Reader
  '#{FILE_PATH_SRC}#'	Absolute file path	OSS Reader HDFS Reader FTP Reader TOS Reader COS Reader S3 Reader Azure Blob Reader

• Edit source columns: You can click Manually Edit Mapping to perform the following operations:

  • Use functions supported by the source database to process columns. For example, use Max(id) to synchronize only the maximum value.

  • Manually edit source columns if the column mapping does not retrieve all columns.

  Note

  MaxCompute Reader does not support the use of functions.

For a single-table batch synchronization task with periodic scheduling, you must configure the properties for automatic scheduling. Go to the editing page of the node, click Scheduling Settings on the right side, and configure the schedule settings for the node.

You must configure scheduling parameters, scheduling policies, scheduling time, and dependencies for the synchronization task. The configuration method is the same as that for other data development nodes and is not described here.

• For schedule settings in the new version of Data Studio, see Node scheduling (new version).

• For schedule settings in legacy Data Studio, see Node scheduling (legacy).

For more information about using scheduling parameters, see Typical scenarios of scheduling parameters in Data Integration.

• Configure run parameters.

  On the right side of the single-table batch synchronization task configuration page, click Run Configuration and configure the following parameters for test runs.

  Configuration item	Description
  Resource Group	Select a resource group that has network connectivity with the data sources.
  Script Parameters	Assign values to the placeholder parameters in the data synchronization task. For example, if the Data Integration task uses the ${bizdate} parameter, configure a date parameter in the yyyymmdd format.

• Run the task.

  Click the Run button on the toolbar to run and debug the task in Data Studio. You can then create a node of the corresponding destination table type to query the destination table data and verify whether the synchronized data meets expectations.

• Deploy the task.

  After the task passes the test run, if the task needs to be run periodically, click the button at the top of the node editing page to deploy the task to the production environment. For more information about task deployment, see Deploy tasks.