DataWorks:Overview

Limits

• The Extensions module is in public preview for users of DataWorks Enterprise Edition and is available in the following regions: China (Beijing), China (Hangzhou), China (Shanghai), China (Zhangjiakou), China (Shenzhen), China (Chengdu), US (Silicon Valley), US (Virginia), Germany (Frankfurt), Japan (Tokyo), China (Hong Kong), and Singapore.
• Only an Alibaba Cloud account or a RAM user to which the AliyunDataWorksFullAccess policy is attached can be used to configure parameters to manage extension point events and processes in the DataWorks console. For more information about permission control by fine-grained RAM policies, see Manage permissions on the DataWorks services and the entities in the DataWorks console by using RAM policies.

Precautions

• If your DataWorks Enterprise Edition expires or extensions are unpublished or disabled, the extensions become invalid, and the checks and blocks performed by the extensions on related processes do not take effect and the related processes directly pass the checks.
• If an extension is in the Testing or Published state, the extension cannot automatically take effect. To make the extension take effect, the workspace administrator must go to the Extension page in SettingCenter to enable the extension.
  • If an extension is in the Testing state, the extension can be enabled only in test workspaces.
  • If an extension is in the Published state, the extension can be enabled in all workspaces in the current region.
• Before you publish an extension, you must commit it to DataWorks for review. After the extension passes the review, the extension is published.
  Note In most cases, an extension review can be complete within T+3 business days after you commit the extension. T indicates the point in time at which you commit the extension for review.

Features

DataWorks provides multiple built-in extension points to support end-to-end big data development and governance. For example, in a workspace in standard mode, after you create a node and develop the code for the node, you must commit the node to the development environment for verification before it can be deployed to the production environment for scheduling. If you no longer need the node, you can delete the node. This process consists of several built-in extension points. You can add custom extension points based on your business requirements. Examples:

• Add an extension point for code review:

  After you click the Submit icon to commit a node to the development environment, the node code is reviewed based on the extension point for code review. The node is committed to the development environment only after the node code passes the review.

• Add an extension point for node review:

  After you click the Deploy icon to deploy a node to the production environment, the node is reviewed based on the extension point for node review. The node is deployed to the production environment only after the node passes the review.

The preceding examples show that you can use file commit and file deployment, which are built-in extension points, and custom extension points in the extension that you registered in DataWorks to perform the process check. You can use the Extensions module to register your local program as an extension and use the OpenEvent and OpenAPI modules to establish a connection between the extension and DataWorks to manage extension point events and processes. The following descriptions provide you with details about how the modules cooperate with each other:

1. OpenEvent: allows you to subscribe to extension point events.

   The OpenEvent module allows you to subscribe to extension point events by using EventBridge. For more information, see Overview. After you receive a message, you can obtain the details of the corresponding event. For example, if you click the Submit icon to commit a node, a file commit event is triggered, and you can receive a message that contains the following information: node ID, operator UID, and workspace ID. For more information about the message formats for different types of events, see Appendix: Formats of event messages sent to EventBridge.

2. Extensions: allows you to process extension point events.

   The Extensions module allows you to register programs as DataWorks extensions. After an extension receives a message, the extension processes the message and calls an API operation to send the processing result to DataWorks. For more information about the extension point events that can be processed by DataWorks extensions, see Supported extension point events.

3. OpenAPI: allows you to call an API operation to send the processing result of an extension point event to DataWorks.
   When an extension point event is triggered and processed by an extension, the event enters the Checking state. After the event is checked, the extension calls an API operation to send the processing result to DataWorks for reference in the subsequent data processing.

   • If the event passes the check, the event status changes from Checking to Check Passed.
   • If the event does not pass the check, the process is blocked and the event status changes from Checking to Check Failed.

Supported extension point events

Procedure