DataWorks:Single-table real-time synchronization task configuration

1. Configure the source

1. On the Configuration tab, select the Kafka topic to synchronize.

   You can use default values for other configurations or modify them as needed. For more information, see the official Kafka documentation .

2. In the upper-right corner, click Data Sampling.

   In the dialog box that appears, set the Start time and Sampled Data Records, and then click Start Collection. This samples data from the specified Kafka topic. You can preview the sampled data, which serves as input for the preview and visualization configurations of subsequent data processing nodes.

3. On the Configure Output Field tab, select the fields to synchronize.

   By default, Kafka provides six fields.

   Field Name	Description
   __key__	The key of the Kafka record.
   __value__	The value of the Kafka record.
   __partition__	The partition number where the Kafka record is located. The partition number is an integer that starts from 0.
   __headers__	The headers of the Kafka record.
   __offset__	The offset of the Kafka record in its partition. The offset is an integer that starts from 0.
   __timestamp__	The 13-digit UNIX timestamp in milliseconds for the Kafka record.

   You can also perform more field transformations in subsequent data processing nodes.

2. Data processing

Enable the Data Processing toggle. Five data processing methods are available: data masking, string replace, data filtering, JSON parsing, and edit and assign fields. You can arrange these methods in any order. At runtime, they are executed sequentially.

After configuring a data processing node, click Preview Data Output in the upper-right corner:

1. The table below the input data shows the results from the previous Data Sampling step. Click Re-obtain Output of Ancestor Node to refresh the results.

2. If there is no upstream output, you can use Manually Construct Data to simulate the previous output.

3. Click Preview to view the processed output from the data processing component.

The Input Data table displays Kafka message fields such as key, value, partition, offset, and timestamp. The Preview Result section shows the parsed fields after data processing, such as key-value pairs from JSON parsing. It also indicates the number of dirty data records and includes a disclaimer: "The preview shown here is for reference only. The results of the actual task execution apply."

3. Configure the destination

1. In the Destination section, select a Tunnel resource group. The default selection is "Public Transport Resources", which corresponds to the free quota for MaxCompute.

2. Specify whether to write to a new table or use an existing table.

   1. If you create a new table, you can select Create from the drop-down list. By default, a table with the same structure as the source is automatically created. You can manually change the destination table name and schema.

   2. If you choose to use an existing table, select the destination table from the drop-down list.

3. (Optional) Edit the table schema.

   Click the edit icon next to the table name to modify its schema. You can click Re-generate Table Schema Based on Output Column of Ancestor Node to automatically generate the schema based on the output columns of the upstream node. You can then select a column in the auto-generated schema to be the primary key.

The sync task provides advanced parameters for fine-grained configuration. Default values are provided and do not need to be changed in most cases. To modify them:

1. In the upper-right corner of the page, click Advanced Settings to go to the Advanced Parameters configuration page.

   Note

   In Data Development, advanced configuration is on a tab on the right side of the task configuration page.

2. You can set parameters separately for the read and write ends of the sync task. Set Automatically set runtime configuration to false to customize the Runtime Configuration.

3. Modify the parameter values based on the tooltips. The description for each parameter is displayed next to its name. For configuration suggestions for some parameters, see Advanced parameters for real-time synchronization.

1. Add an alert

Select Use custom rules, and enter an Alert Name and Description. Supported notification methods include Email, SMS, Phone, DingTalk, webhook, and Feishu. You can configure these methods independently for WARNING and CRITICAL levels. Recipients can be specified through an on-call schedule or added manually.

(1) Click Create Rule to configure an alert rule.

You can set the Alert Reason to monitor metrics such as Business delay, failover, Task status, DDL Notification, and Task Resource Utilization. You can then set CRITICAL or WARNING alert levels based on specified thresholds.

• After setting the alert method, use Configure Advanced Parameters to control the interval for sending alert notifications, which prevents message flooding.

• If you select Business delay, Task status, or Task Resource Utilization as the alert cause, you can also enable recovery notifications to inform recipients when the task returns to a normal state.

(2) Manage alert rules.

For existing alert rules, you can use the toggle to enable or disable them. You can also send alerts to different personnel based on the alert level.

2. View alerts

In the task list, expand More > Configure Alert Rule to go to the alert events page and view historical alerts.