Data Integration supports real-time synchronization from single-table sources such as Kafka and LogHub to OSS. This example uses Kafka as the source and OSS as the destination.
Limitations
The Kafka version must be between 0.10.2 and 2.2.0 (inclusive).
Prerequisites
-
You have purchased a serverless resource group or an exclusive resource group for Data Integration.
-
You have created a Kafka data source and an OSS data source. For more information, see Data source configuration.
-
You have established network connectivity between the resource group and the data sources. For more information, see Overview of network connectivity solutions.
Procedure
Step 1: Select a synchronization task type
Log on to the DataWorks console. In the target region, click in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Integration.
-
In the left-side navigation pane, click Synchronization Task. Then, click Create Synchronization Task at the top of the page. On the task creation page, set the following parameters:
-
Source and destination:
Kafka→OSS -
New Node Name: Enter a custom name for the synchronization task.
-
Synchronization Method:
single-table real-time.
-
Step 2: Configure network and resources
-
In the Network and Resource Configuration section, select the Resource Group for the synchronization task. You can allocate Task Resource Usage in CUs for the task.
-
For Source Information, select your
kafkadata source. For Destination, select yourOSSdata source. Then click Test Connectivity. -
After both the source and destination data sources pass the connectivity test, click Next.
Step 3: Configure the synchronization link
1. Configure the Kafka source
At the top of the page, click the Kafka source node to configure the Source Information.

-
In the Source Information section, select the topic to synchronize from the Kafka cluster.
You can use the default values for other parameters or modify them as needed.
-
In the upper-right corner, click Data Sampling.
In the dialog box that appears, specify the Start Time and Number of Sample Records, and then click Start Sampling. This allows you to sample and preview data from the specified Kafka topic. The sampled data provides input for data previews and visual configurations in subsequent data processing nodes.
-
In the Output Field Configuration section, select the fields to synchronize.
2. Edit data processing nodes
Click the
icon to add a data processing method. Five data processing methods are available: data masking, string replacement, data filtering, JSON parsing, and field editing and assignment. You can arrange these methods in any order. When the task runs, it processes data sequentially in the configured order.
After you configure each processing node, you can click Output Preview in the upper-right corner. In the dialog box, click Retrieve Upstream Output to simulate how the current node processes the sampled data and view the output.
The Preview Data Output dialog box has two sections: Input Data and Preview Result. Input Data displays the sampled Kafka data retrieved by clicking Re-obtain Output Of Ancestor Node. You can also click + Manually Add Data to add test data. The table includes fields such as _key_, _value_, _partition_, _offset_, and timestamp. In Preview Result, click Preview to view the processed output and dirty data count. The preview result is for reference only and may differ from actual task execution.
The data output preview feature relies on the Data Sampling of the Kafka source. You must perform data sampling in the Kafka source configuration before you preview the output.
3. Configure the OSS destination
At the top of the page, click the OSS destination node to configure the OSS Destination Information.
-
In the OSS Destination Information section, configure the basic information for the OSS destination.
-
Write Format: Supported formats include Hudi, Paimon, and Iceberg.
-
Location for Creating Metadatabase: If you have activated Data Lake Formation (DLF) for your account, the system automatically creates the corresponding metadatabase and metatables in DLF when data is synchronized to the data lake.
NoteCross-region metadata creation is not supported.
-
Select Storage Path: Select the OSS path to store the data in the data lake.
-
Target Database: Select the destination database for the data. You can also select Create Database to create a DLF metadatabase and specify a Database Name.
-
Destination Table: Choose whether to Create tables automatically automatically or Use Existing Table to write data to an OSS table.
-
Table Name: Enter or select the name of the OSS table to write to.
-
-
(Optional) Edit the table schema.
When you select Create tables automatically, click the Edit Table Schema button to edit the target table structure in the dialog box. You can also click Regenerate Schema from Upstream Node to automatically generate the table structure from the upstream node's output columns. In the generated structure, you can select a column to set as the primary key.
-
Configure field mapping.
The system automatically generates mappings between upstream columns and target columns based on the The same name mapping principle. You can adjust these mappings as needed. A single upstream column can be mapped to multiple target columns, but multiple upstream columns cannot be mapped to a single target column. If an upstream column is not mapped to a target column, its data will not be written to the target table.
4. Alerts
To prevent task errors from causing delays in business data synchronization, you can set an alert policy for the synchronization task.
-
Click Alert Settings in the upper-right corner of the page to open the Alert Rule Configurations for Real-time Synchronization Subnode settings page.
-
Click Add Alert Rule to configure an alert rule.
NoteThe alert rules you define here apply to the real-time synchronization subtasks that this task generates. After you configure the task, you can view and modify the alert rules for these subtasks on the Run and manage real-time synchronization tasks page.
-
Manage alert rules.
For existing alert rules, you can use the toggle switch to enable or disable them. You can also send alerts to different recipients based on the alert level.
5. Advanced settings
The synchronization task provides several parameters that you can modify as needed.
Before making changes, ensure that you fully understand the function of each parameter to prevent unexpected errors or data quality issues.
-
Click advanced settings in the upper-right corner of the page to open the advanced settings page.
-
On the advanced settings page, modify the parameter values as needed.
Step 6: Configure DDL capability
The source data source may contain various DDL operations. Based on your business requirements, you can click DDL Capability Configuration in the upper-right corner of the page to go to the DDL capability configuration page and define processing policies for different DDL messages to be synchronized to the destination.
For more information about different DDL message processing policies, see DDL message processing rules.
Step 7: Configure a resource group
You can click Configure Resource Group in the upper-right corner to view and switch the task's current resource group.
Step 8: Perform a simulated run
After you configure the task, click Dry Run in the upper-right corner. This feature simulates the entire task on a small data sample and allows you to preview the results in the target table. If there are configuration errors, runtime exceptions, or dirty data, you will receive real-time error messages. This helps you quickly verify that the task is configured correctly and produces the expected results.
-
In the dialog box, set the sampling parameters: Start Time and Number of Sample Records.
-
Click Start Sampling to collect the sample data.
-
Click Preview to simulate the entire task processing using the sampled data.
Step 9: Run the synchronization task
-
After completing all settings, click Complete at the bottom of the page.
-
On the page, find the task you created and click Start in the Operations column.
-
Click the Name/ID of the corresponding task in the Task List to view its detailed execution process.
Manage the synchronization task
View task status
After you create a synchronization task, you can view task status and details on the synchronization task page.
-
In the Start column, you can Stop or Stop a task. Additional operations, such as Edit and View, are available under the "More" option.
-
For a running task, you can check its general status in the Execution Overview column or click the overview area to view detailed execution information.
A Kafka-to-OSS single-table real-time synchronization task includes two stages:
-
Schema Migration: Shows how the destination table was created (from an existing table or by automatic creation). If you select to automatically create the table, the system displays the DDL statement used.
-
Real-time Data Synchronization: Displays statistics about the real-time synchronization, including runtime information, DDL records, and alerts.
Rerun a task
In special cases, such as when you need to modify synchronized fields or adjust target table information, you can click Rerun in the Operations column of the synchronization task. This action synchronizes the adjusted fields and other changes to the target. The process skips unchanged, previously synchronized tables.
-
To run the task again without any changes, click Rerun.
-
If you edit the task, click Complete after making your changes. The task's action changes to Apply Updates. Clicking Apply Updates reruns the task with the new configuration.