The OpenEvent module sends DataWorks event messages to a shared Message Queue for Apache Kafka topic. You can subscribe to the topic to receive the messages. Before you can use the features of the OpenEvent module, you must enable message subscription and set related parameters in the DataWorks console. This topic describes how to enable message subscription and configure a Message Queue for Apache Kafka topic.

Background information

DataWorks publishes change event messages to a Kafka topic. Consumers subscribe to the topic to receive the messages. For more information about the logic for message subscription, see Overview.

DataWorks is integrated with Message Queue for Apache Kafka. After you enable message subscription in the DataWorks console, DataWorks automatically generates a topic, a consumer group, and a user for you. You only need to specify the workspaces and event types to which the topic applies.
Note For more information about topics, consumer groups, and users, see Terms.

Limits

  • Only DataWorks Enterprise Edition users can join the public preview program by submit a ticket . No fees are required for the program.
  • This module is in public preview only in the China (Beijing), China (Hangzhou), China (Shenzhen), and China (Shanghai) regions.
  • You can complete the configuration only by using your Alibaba Cloud account or as a RAM user that has the AliyunDataWorksFullAccess permission.
  • After you enable the message subscription feature, DataWorks automatically generates a topic and a consumer group for you. You can also create consumer groups as needed. You can create five consumer groups at most during the public preview.
    Note For more information about topics, consumer groups, and users, see Terms .

Step 1: Enable the message subscription feature

  1. Log on to the DataWorks console. In the left-side navigation pane, click Open Platform.
  2. Click the OpenEvent tab.
  3. Turn on Enable Message Subscription.
    Note After you enable the message subscription feature, DataWorks automatically creates a topic and pushes messages of the specified event types to the topic. If you turn off Enable Message Subscription, DataWorks stops pushing messages to the topic.
    Enable Message Subscription
After you enable the message subscription feature, DataWorks automatically creates a Kafka user, a Kafka topic, and a consumer group for you.

Step 2: Configure the Kafka topic

After you enable the message subscription feature, DataWorks automatically creates a Kafka user, a Kafka topic, and a consumer group for you. You only need to specify the workspaces and event types to which the topic applies. To configure the Kafka topic, perform the following steps:

  1. In the Kafka Topic Information section, click Configure Subscription in the Actions column.
  2. In the dialog box that appears, select the workspaces and event types as needed.
    Topic SettingsDataWorks allows you to subscribe to the following types of events: file change events, instance status change events, and table change events. For more information about supported event types, see Supported event types.

View the Kafka configurations

After you enable the message subscription feature and configure the Kafka topic, you can develop consumer programs on your applications based on the Kafka configurations. You can view the Kafka configurations in the DataWorks console. View the Kafka configurations

What to do next

After you enable the message subscription feature and configure the Kafka topic, you can obtain the Kafka configurations and develop consumer programs by using Message Queue for Apache Kafka SDK. This way, you can consume the messages published by DataWorks. For more information about how to create a consumer program, see Develop consumer programs on your clients.