DataWorks allows you to use the Extensions module to register programs as DataWorks extensions. You can use the Extensions module together with the message subscription feature provided by the OpenEvent module to manage extension point events and processes. You can also customize configurations based on your business requirements. This topic provides an overview of the Extensions module.

Features

DataWorks provides multiple built-in processes to support end-to-end big data development and governance. For example, in a workspace in standard mode, after you create a node and develop the code for the node, you must commit the node to the development environment for verification before it can be deployed to the production environment for scheduling. If you no longer need the node, you can delete the node. FeaturesSeveral built-in extension points are involved in this process. You can add custom extensions to improve the process and facilitate data management. Examples:
  • Add an extension for code review:

    After you click the Submit icon to commit a node to the development environment, the node code is reviewed by the extension, and the node can be committed only after the code is approved by the extension.

  • Add an extension for node review:

    After you click the Deploy icon to deploy a node to the production environment, the node is reviewed by the extension, and the node can be deployed only after it is approved by the extension.

The preceding examples show that the Extensions module can work with the OpenEvent and OpenAPI modules to subscribe to and respond to extension point events. The following section describes how the partnership is accomplished:
  1. OpenEvent: subscribes to DataStudio extension point events.

    The OpenEvent module provides a shared Message Queue for Apache Kafka topic for you to subscribe to extension point events. For more information, see Overview. After you receive a message, you can obtain the details of the corresponding event. For example, after you click the Submit icon to commit a node, a file commitment event is triggered, and you can receive a message that contains the following information: the node ID, the operator UID, and the workspace ID. For more information about the message formats for different types of extension point events, see Appendix: Message formats.

    You can also call the GetIDEEventDetail operation to obtain the following details of the node based on the message ID: the dependencies between the node and other nodes, the node code, and the scheduling configurations of the node.

  2. Extensions: checks DataStudio extension point events.

    The Extensions module allows you to register programs as DataWorks extensions. After an extension receives a message, the extension checks the message and calls an API operation to send the check result to DataWorks. For more information about the extension point events that can be checked by DataWorks extensions, see Supported extension point events.

  3. OpenAPI: sends the check results of DataStudio extension point events to DataWorks.
    When an extension point event is triggered and checked by an extension, the current process is suspended and enters the Checking state. After the event is checked, the extension calls an API operation to send the check result to DataWorks for reference in the subsequent data processing. The API operation that is called to send the check result varies based on modules. For more information, see Supported extension point events.
    • If the event passes the check, the current process is restored and enters the Check Passed state.
    • If the event does not pass the check, the current process is blocked and enters the Check Failed state.

Supported extension point events

The Extensions module supports only part of the DataStudio extension point events. The following table describes supported extension point events.
DataWorks service Extension point Extension point event API operation for sending check results
DataStudio Extension point for file commitment File commitment events: events that are triggered when you commit nodes, resources, or functions.
Note This extension point is supported only in workspaces in standard mode.
UpdateIDEEventResult
Extension point for file deployment File deployment events: events that are triggered when you deploy nodes, resources, or functions.
Extension point for file execution Code execution events: events that are triggered when you execute the code of nodes.
Extension point for file deletion File deletion events: events that are triggered when you delete nodes, resources, or functions.
Extension point for table commitment Table commitment events: events that are triggered when you commit tables on the Workspace Tables page of DataStudio.
Note This extension point is supported only in workspaces in standard mode.
Extension point for table deployment Table deployment events: events that are triggered when you deploy tables on the Workspace Tables page of DataStudio.
Operation Center Extension point for node freezing Node freezing events: events that are triggered when you freeze auto triggered nodes. UpdateWorkbenchEventResult
Extension point for node unfreezing Node unfreezing events: events that are triggered when you unfreeze auto triggered nodes.
Extension point for node undeployment Node undeployment events: events that are triggered when you undeploy nodes in the production environment.
Extension point for data backfill Data backfill events: events that are triggered when you perform data backfill operations on auto triggered nodes. For example, a data backfill event is triggered when you backfill data for the current node, backfill data for the current node and its descendant nodes, backfill data for a large number of nodes, or backfill data in advanced mode.
The following figure shows the check process of a node commitment event. File commitmentFor more information about the check processes for various types of extension point events, see Trigger event checking during data development and Trigger event checking in Operation Center.

Procedure

To implement custom management of processes by using the OpenAPI, OpenEvent, and Extensions modules, perform the following steps:
  1. Make preparations.

    Enable the message subscription feature, register an extension, and obtain the information required to develop the extension. For more information, see Make preparations.

  2. Develop and deploy the extension.

    Develop the extension and deploy the extension as a service application. For more information, see Develop and deploy an extension.

  3. Test and enable the extension.

    After the extension is developed and deployed, test whether the extension performs as expected in a test workspace. If the extension passes the test, publish the extension. Then, workspace administrators can enable the extension for workspaces as needed. For more information, see Test and enable an extension.