DataWorks:Batch synchronization from a single Kafka table to MaxCompute

Step 1: Create a node

Follow the Codeless UI configuration guide to create and configure an offline synchronization node. This tutorial focuses on the configuration details specific to Kafka-to-MaxCompute synchronization.

Step 2: Configure the data source and destination

Step 3: Configure field mapping

After selecting the source and destination, map columns between the Kafka reader and the MaxCompute writer. Use Map Fields with Same Name, Map Fields in Same Line, Clear Mappings, or Manually Edit Mapping as needed.

Default Kafka fields

Kafka exposes six built-in fields that you can map directly to MaxCompute columns.

JSON field parsing

For JSON-formatted Kafka values, add custom field definitions using . to access subfields and [] to access array elements.

Given this example record value:

{
  "a": {
    "a1": "hello"
  },
  "b": "world",
  "c": [
    "xxxxxxx",
    "yyyyyyy"
  ],
  "d": [
    { "AA": "this", "BB": "is_data" },
    { "AA": "that", "BB": "is_also_data" }
  ],
  "a.b": "unreachable"
}

Mapping rules

• Unmapped source fields are not read by the synchronization instance.

• NULL is written to unmapped destination fields.

• One source field cannot map to multiple destination fields.

• One destination field cannot be mapped from multiple source fields.

Step 4: Configure advanced parameters

Click Advanced Configuration on the right side of the task. For this tutorial, set Policy for Dirty Data Records to Ignore Dirty Data Records and leave all other parameters at their defaults. For parameter details, see Codeless UI configuration.

Step 5: Test the synchronization

1. Click Run Configuration on the right side of the editing page. Set the Resource Group and Script Parameters for the test run, then click Run in the top toolbar.

2. After the test run completes, verify the data in the destination table. In the left navigation pane, create a file with the .sql extension and run the following query:

   Note

   • To query data this way, attach the destination MaxCompute project to DataWorks as a computing resource.

   • On the .sql file editing page, click Run Configuration on the right, specify the Computing Resource, and Resource Group, then click Run.

   SELECT * FROM <MaxCompute_destination_table_name> WHERE pt=<specified_partition> LIMIT 20;

Step 6: Configure scheduling and publish

Click Scheduling on the right side of the task to configure the scheduling settings for the recurring run, then click Publish to publish the task.

The three scheduling parameters used in this tutorial—${startTime}, ${endTime}, and ${partition}—are linked: the values injected into ${startTime} and ${endTime} control the Kafka time window for each instance, while ${partition} determines which MaxCompute partition receives the data. Configure all three together based on your scheduling cycle.

The following examples cover the most common scheduling patterns.