DataWorks:Configure a real-time synchronization task in Data Integration

1. Configure the source

At the top of the page, click the Kafka data source. Then, edit the Kafka Source Information.

1. In the Kafka Source Information section, select the topic to synchronize from the Kafka data source.

   You can use the default values for other configurations or modify them as needed. For more information about the parameters, see the official Kafka documentation .

2. In the upper-right corner, click Data Sampling.

   In the dialog box that appears, set the Start Time and Number Of Samples, and then click Start Sampling. This action samples data from the specified Kafka topic. You can preview the data in the topic. This preview provides input for the data preview and visualization configurations of subsequent data processing nodes.

3. In the Output Field Configuration section, select the fields to synchronize as needed.

   By default, Kafka provides six fields.

   Field Name	Description
   __key__	The key of the Kafka record.
   __value__	The value of the Kafka record.
   __partition__	The partition number where the Kafka record is located. The partition number is an integer that starts from 0.
   __headers__	The headers of the Kafka record.
   __offset__	The offset of the Kafka record in its partition. The offset is an integer that starts from 0.
   __timestamp__	The 13-digit UNIX timestamp in milliseconds for the Kafka record.

   You can also perform more field transformations in subsequent data processing nodes.

2. Edit the data processing node

Click the icon to add a data processing method. Five methods are available: Data Masking, String Replace, Data Filtering, JSON Parsing, and Edit and Assign Fields. You can arrange these methods in the desired order. At runtime, the data processing methods are executed in the specified order.

After you configure a data processing node, you can click Preview Data Output in the upper-right corner:

1. In the table below the input data, you can view the results from the previous Data Sampling step. Click Re-fetch Upstream Output to refresh the results.

2. If there is no output from the upstream node, you can also click Manually Construct Data to simulate the previous output.

3. Click Preview to view the output data from the upstream step after it is processed by the data processing component.

3. Configure the destination

At the top of the page, click the MaxCompute data destination. Then, edit the MaxCompute destination information.

1. In the MaxCompute Destination Information section, select a Tunnel resource group. The default is "Public transport resources", which is the free quota provided by MaxCompute.

2. Specify whether to Auto-create Table or Use Existing Table for the destination table.

   1. If you choose to auto-create a table, a table with the same name as the source table is created by default. You can manually change the destination table name.

   2. If you choose to use an existing table, select the destination table from the drop-down list.

3. (Optional) Edit the table schema.

   When you select Auto-create Table, click Edit Table Schema. In the dialog box that appears, edit the destination table schema. You can also click Regenerate Table Schema Based On Upstream Output Columns to automatically generate the table schema based on the output columns of the upstream node. You can select a column in the auto-generated schema and set it as the primary key.

4. Configure field mapping.

   1. The system automatically maps upstream columns to destination table columns based on the Same Name Mapping principle. You can adjust the mappings as needed. An upstream column can be mapped to multiple destination columns, but multiple upstream columns cannot be mapped to a single destination column. If an upstream column is not mapped to a destination column, its data is not written to the destination table.

   2. For Kafka fields, you can configure custom JSON parsing. Use the data processing component to retrieve the content of the value field. This allows for more fine-grained field configuration.

      

5. (Optional) Configure partitions.

   1. Automatic Time-based Partitioning creates partitions based on the business time (in this case, the _timestamp field). The first-level partition is by year, the second-level partition is by month, and so on.

   2. Dynamic Partitioning by Field Content maps a field from the source table to a partition field in the destination MaxCompute table. This ensures that rows containing specific data in the source field are written to the corresponding partition in the MaxCompute table.

Alert configuration

To prevent data synchronization latency caused by task errors, you can set an alert policy for the single-table real-time synchronization task.

1. In the upper-right corner of the page, click Alert Settings to open the alert settings page for the task.

2. Click Add Alert to configure an alert rule. You can set alert triggers to monitor metrics such as data latency, failover events, task status, Data Definition Language (DDL) changes, and task resource utilization. You can set CRITICAL or WARNING alert levels based on specified thresholds.

3. Manage alert rules.

   For existing alert rules, you can use the alert switch to enable or disable them. You can also send alerts to different personnel based on the alert level.

Advanced parameter configuration

The sync task provides advanced parameters for fine-grained configuration. The system provides default values, which you do not need to change in most cases. To modify them:

1. In the upper-right corner of the page, click Advanced Parameters to open the advanced parameter configuration page.

2. Set Auto-configure Runtime Settings to false.

3. Modify the parameter values based on the tooltips. The description for each parameter is displayed next to its name.

Resource group configuration

In the upper-right corner of the page, you can click Resource Group Configuration to view and switch the resource group used by the current task.