The OpenEvent module allows you to subscribe to event messages so that you can detect various DataWorks change events and respond to the events at the earliest opportunity. For example, you can detect changes to core tables in real time by subscribing to table change events. You can also customize a dashboard to display the status of nodes in real time by subscribing to node change events. This topic describes how the OpenEvent module works and how to configure message subscription.

Limits

The OpenEvent module is in public preview.
  • Only DataWorks Enterprise Edition users can join the public preview program by submit a ticket . No fees are required for the program.
  • This module is in public preview only in the China (Beijing), China (Hangzhou), China (Shenzhen), China (Shanghai) and China (Hong Kong) regions.

Supported event types

DataWorks allows you to subscribe to both regular events and extension point events.
  • Regular events: For this type of event, you can only subscribe to them.
  • Extension point events: For this type of event, you can not only subscribe to them but also configure extensions to respond to them.
You can subscribe to the most important events in Operation Center and DataStudio. The following table describes supported event types.
DataWorks service Event category Event description and type Example
Operation Center Instance status change events
  • Description: the events that are triggered by the changes to the status of instances.
  • Type: regular events.
An instance status change event is triggered when the status of an instance changes from Not running to Running or from Running to Run failed. Flowchart
Node change events
  • Description: the events that are triggered by the changes to the status of nodes in the production environment in Operation Center after the nodes are managed, committed, and deployed in DataStudio.
  • Type: regular events.
A node change event is triggered when a node is created, updated, or deleted in the production environment in Operation Center after the node is managed, committed, and deployed in DataStudio.
DataStudio File change events
  • Description: the events that are triggered by the changes to the status of files such as nodes, resources, and functions.
  • Type: extension point events
  • File commitment events:

    A file commitment event is triggered when a node, resource, or function is committed.

  • File deployment events:

    A file deployment event is triggered when a node, resource, or function is deployed.

  • Node execution events:

    A node execution event is triggered when the code of a node is executed.

  • File deletion events:

    A file deletion event is triggered when a node, resource, or function is deleted.

Table change events
  • Description: the events that are triggered by the changes to the status of tables.
  • Type: extension point events
  • Table commitment events: A table commitment event is triggered when a table is committed on the Workspace Tables page of DataStudio.
  • Table deployment events: A table deployment event is triggered when a table is deployed on the Workspace Tables page of DataStudio.
    Note You can deploy tables on the Workspace Tables page only when the workspace is in standard mode. That is, extension point events related to table deployment can be triggered only in workspaces that are in standard mode.
Note Network latency may lead to a delay in delivering OpenEvent messages.

Logic for message subscription

The OpenEvent module of DataWorks implements message subscription based on Message Queue for Apache Kafka. By default, DataWorks provides a shared Message Queue for Apache Kafka topic for you to subscribe to DataWorks change events. You do not need to purchase Apache Kafka yourself. OpenEventThe preceding figure shows the logic for message subscription.
  • DataWorks serves as a producer, which publishes messages related to DataWorks change evens to a Kafka topic. A client serves as a consumer, which subscribes to the Kafka topic to receive messages. By default, DataWorks provides a shared Message Queue for Apache Kafka topic for you to subscribe to DataWorks change events. You only need to configure the topic.
  • When you configure the topic, you must specify the workspaces and event types to which the topic applies. After the configuration is complete, if events of the specified types are triggered in the specified workspaces, DataWorks automatically publishes messages to the topic for clients to consume.
    Note For more information about supported event types, see Supported event types.
  • DataWorks and clients can publish and receive messages by using the SSL endpoints of Message Queue for Apache Kafka. You can view the SSL endpoints on the OpenEvent tab of the Open Platform page in the DataWorks console.
  • If a client subscribes to messages by using an SSL endpoint, the PLAIN mechanism is used for identity authentication. The PLAIN mechanism works based on a username and a password. The OpenEvent module automatically generates a username and a password when you enable message subscription. You can also delete the default user and create new user information for message subscription.

Main configuration steps

To use the features of the OpenEvent module, you must complete the following configuration steps:

  1. Enable message subscription in DataWorks.

    In DataWorks, you must enable the message subscription feature, configure the username and password for identity authentication, and specify the workspaces and event types to which the topic applies. For more information, see Enable message subscription in DataWorks.

  2. Develop consumer programs on your clients.

    After the configuration in DataWorks is complete, you can develop consumer programs on your clients to consume messages. For more information about the sample code, see Develop consumer programs on your clients. For information about the message formats for different types of DataWorks events, see Appendix: Message formats.

Benefits

The OpenEvent module provides the following benefits:
  • Lower O&M costs:

    You can subscribe to important DataWorks events with ease and have free access to Message Queue for Apache Kafka that features data persistence, high throughput, and enterprise-level data security protection.

  • More efficient message publishing and receiving:

    You only need to configure the Kafka topic in DataWorks and develop consumer programs on your clients. You do not need to develop producer programs for DataWorks. This reduces and simplifies manual operations.

  • Applicable to various needs:

    You can subscribe to various types of DataWorks events. This allows you to detect important DataWorks changes and respond to the changes at the earliest opportunity.