Overview - DataWorks - Alibaba Cloud Documentation Center

You can use the OpenEvent module of DataWorks Open Platform to subscribe to event messages that are generated in your DataWorks workspace. You can use the Extensions module of DataWorks Open Platform to register your local program as an extension to manage extension point events and processes. This topic describes how the Extensions module works with the OpenEvent module to manage extension point events and processes.

Limits

The Extensions module is in public preview for users of DataWorks Enterprise Edition and is available in the following regions: China (Beijing), China (Hangzhou), China (Shanghai), China (Zhangjiakou), China (Shenzhen), China (Chengdu), US (Silicon Valley), US (Virginia), Germany (Frankfurt), Japan (Tokyo), China (Hong Kong), and Singapore.
Only an Alibaba Cloud account or a RAM user to which the AliyunDataWorksFullAccess policy is attached can be used to configure parameters to manage extension point events and processes in the DataWorks console. For more information about permission control by fine-grained RAM policies, see Manage permissions on the DataWorks services and the entities in the DataWorks console by using RAM policies.

Features

DataWorks provides multiple built-in extension points to support end-to-end big data development and governance. For example, in a workspace in standard mode, after you create a node and develop the code for the node, you must commit the node to the development environment for verification before it can be deployed to the production environment for scheduling. If you no longer need the node, you can delete the node. Features

This process consists of several built-in extension points. You can add custom extension points based on your business requirements. Examples:

Add an extension point for code review:
After you click the Submit icon to commit a node to the development environment, the node code is reviewed based on the extension point for code review. The node is committed to the development environment only after the node code passes the review.
Add an extension point for node review:
After you click the Deploy icon to deploy a node to the production environment, the node is reviewed based on the extension point for node review. The node is deployed to the production environment only after the node passes the review.

The preceding examples show that you can use file commit and file deployment, which are built-in extension points, and custom extension points in the extension that you registered in DataWorks to perform the process check. You can use the Extensions module to register your local program as an extension and use the OpenEvent and OpenAPI modules to establish a connection between the extension and DataWorks to manage extension point events and processes. The following descriptions provide you with details about how the modules cooperate with each other:

OpenEvent: allows you to subscribe to extension point events.
The OpenEvent module provides a shared Message Queue for EventBridge topic for you to subscribe to extension point events. For more information, see Overview. After you receive a message, you can obtain the details of the corresponding event. For example, if you click the Submit icon to commit a node, a file commit event is triggered, and you can receive a message that contains the following information: node ID, operator UID, and workspace ID. For more information about the message formats for different types of events, see Appendix: Formats of event messages sent to EventBridge.
Extensions: allows you to process extension point events.
The Extensions module allows you to register programs as DataWorks extensions. After an extension receives a message, the extension processes the message and calls an API operation to send the processing result to DataWorks. For more information about the extension point events that can be processed by DataWorks extensions, see Supported extension point events.
OpenAPI: allows you to call an API operation to send the processing result of an extension point event to DataWorks.
When an extension point event is triggered and processed by an extension, the event enters the Checking state. After the event is checked, the extension calls an API operation to send the processing result to DataWorks for reference in the subsequent data processing.
- If the event passes the check, the event status changes from Checking to Check Passed.
- If the event does not pass the check, the process is blocked and the event status changes from Checking to Check Failed.

Supported extension point events

The following table describes different types of extension point events that can be processed by DataWorks extensions.

Note When you register and develop your local program as a DataWorks extension, you must specify the codes for the extension point events that are processed by the extension based on the extension point event codes provided by DataWorks. For information about the code of each type of extension point event, see Development references: Extension point event codes.


DataWorks module	Extension point	Extension point event	API operation for sending processing results
DataStudio	Extension point for file commit	File commit pre-events: an event that is triggered when you commit a node, resource, or function.	You can call the UpdateIDEEventResult operation to send the processing results of extension point events to DataWorks.
	Extension point for file deployment	File deployment pre-events: an event that is triggered when you deploy a node, resource, or function. Note This type of extension point event is supported only in workspaces in standard mode.
	Extension point for file execution	File execution pre-events: an event that is triggered when you execute the code of a node.
	Extension point for file deletion	File deletion pre-events: an event that is triggered when you delete a node, resource, or function.
	Extension point for table commit	Table commit pre-events: an event that is triggered when you commit a table on the Workspace Tables page of DataStudio.
	Extension point for table deployment	Table deployment pre-events: an event that is triggered when you deploy a table on the Workspace Tables page of DataStudio. Note This type of extension point event is supported only in workspaces in standard mode.
Operation Center	Extension point for node freezing	Node freezing pre-events: an event that is triggered when you freeze a node in Operation Center.	You can call the UpdateWorkbenchEventResult operation to send the processing results of extension point events to DataWorks.
	Extension point for node unfreezing	Node unfreezing pre-events: an event that is triggered when you unfreeze a node in Operation Center.
	Extension point for node undeployment	Node undeployment pre-events: an event that is triggered when you undeploy a node in Operation Center.
	Extension point for data backfill	Data backfill pre-events: an event that is triggered when you backfill data for a node in Operation Center.

The following figure shows the process of committing a node in DataStudio. File commit

For more information about the check processes for various types of extension point events, see Trigger event checking during data development and Trigger event checking in Operation Center.

Procedure

This section describes the operations that you must perform to implement custom management of processes by using the OpenAPI, OpenEvent, and Extensions modules.

Make preparations.
Enable the message subscription feature, register an extension, and obtain the information required to develop the extension. For more information, see Preparations.
Develop and deploy the extension.
Develop the extension and deploy the extension as a service application. For more information, see Develop and deploy a custom extension.
You can configure parameters and define options for an extension. This allows you to use the extension in a flexible and efficient manner. For more information, see Advanced feature: Configure extension parameters and Advanced feature: Define options for an extension.
Test and enable the extension.
After the extension is developed and deployed, test whether the extension performs as expected in a test workspace. If the extension passes the test, publish the extension. Then, workspace administrators can enable the extension for workspaces as needed. For more information, see Test and enable an extension.