DataWorks Extensions is a plug-in that works with OpenAPI and OpenEvent. You can use extensions to implement custom processing logic for user operations in DataWorks and control behaviors such as interception and blocking. For example, you can develop an extension for task change control and use it to implement custom task deployment control. This topic describes the basic information about extensions.
Limits
Only DataWorks Enterprise Edition is supported.
Supported regions: China (Beijing), China (Hangzhou), China (Shanghai), China (Zhangjiakou), China (Shenzhen), China (Chengdu), US (Silicon Valley), US (Virginia), Germany (Frankfurt), Japan (Tokyo), China (Hong Kong), Singapore.
Notes
Access control: Only Open Platform Administrators and Tenant Administrators, Alibaba Cloud account or RAM users with AliyunDataWorksFullAccess permission have read and write permissions for the developer backend. For more information, see Global module permission control and Product and console permission control details: RAM Policy.
Version period: If your DataWorks Enterprise Edition expires, all extensions become invalid and can no longer trigger event checks. If an extension is triggered to check an event and has not completed the check when DataWorks Enterprise Edition expires, the check is terminated and the result Check Passed is returned.
Limits: Currently, only some extension point events support deploying extensions through Function Compute.
Product capabilities
DataWorks Open Platform provides various extension points. The Extensions module is provided based on these extension points. The Extensions module is a plug-in that works with the OpenAPI and OpenEvent modules to provide features that meet your business requirements and help you manage operations in DataWorks based on custom processing logic, such as blocking operations.
Business scenarios in which extensions can be used for operation management:
Management of table or task naming conventions.
Management of duplicate data synchronization tasks.
Management of fees generated by tasks.
Management of dependencies between tasks.
NoteYou can implement more scenarios by referring to the event types in Development reference: Event list and message format.
Custom process control: When you perform an operation in DataWorks and the extension point event generated by the operation is specified in an extension, you cannot continue the operation until the extension returns the processing result of the event message of the extension point event.
For example: In a workspace in standard mode, you can add an extension that checks a specific function to the original task development and publishing process. This transforms the task process from Development > Submit > Publish to Development > Pre-submission verification > Submit > Pre-publishing check > Publish. For an example, see Best practices: (Advanced feature application) Prohibit the use of the MAX_PT function.
Development process
In DataWorks, you can develop and deploy extensions to implement custom process control through self-built services or Function Compute services.
Configure the required settings in OpenEvent to push event messages: Select an appropriate method to develop and deploy extensions based on the types of extension point events that you need to receive for your business.
Develop and deploy extensions using self-built services: This method relies on the message distribution capability of EventBridge. Make sure that you have specified the event bus in EventBridge to which DataWorks messages are sent, and have configured the routing service for messages in the event bus.
Develop and deploy extensions using Function Compute: DataWorks messages are pushed to the specified FC function service by default. You do not need to configure an event bus in OpenEvent.
Develop and deploy a custom extension: Receive and parse event messages pushed by DataWorks in the extension, implement custom logic judgment, and return the results to DataWorks.
Develop and deploy extensions: Self-built service method: Use the service program to which messages are ultimately routed in the event bus to accept DataWorks messages, make logical judgments on the messages, and call back the processing results through the specified API.
Develop and deploy extensions: Function Compute method: You need to implement specific interfaces of Function Compute to receive extension requests and return extension processing results.
Register the extension: Define which types of event messages the developed extension will receive and process on the DataWorks side.
Test the extension: Verify whether the extension meets expectations in a test workspace.
Submit and publish the extension: After confirming that there are no issues, submit the extension for platform review. After the review is approved, the extension is published for all workspaces to use.
NoteThe review of extensions is usually completed within
T+3working days. Please be patient.DataWorks event messages are valid for three days. If the validity period is exceeded, the event messages are considered expired and are not processed by an extension.
Supported extension point events
The following table describes the types and details of extension point events that can be processed by extensions for each DataWorks service.
When developing and deploying extensions, you need to use the eventcode corresponding to different extension points. For details about the event codes of each event, see Development reference: Event list and message format.
You can publish extension point events only in workspaces in standard mode.
For the interception effect of extension points, see Application example: Trigger event checks during data development and Application example: Trigger event checks in Operation Center.
Application scope | System module | Extension point type | Extension point | OpenAPI for callback verification results |
Workspace level | File change events
| Pre-event for file deletion | You can use the UpdateIDEEventResult API to call back the event processing results to DataWorks. | |
Pre-event for file committing | ||||
Pre-event for file publishing | ||||
Pre-event for code running | ||||
Table change events | Pre-event for table submission to production | |||
Pre-event for table submission to development | ||||
Node change events | Enable task | You can use the CallbackExtension API to call back the event processing results to DataWorks. | ||
Batch real-time task | ||||
Node change events | Pre-event for node unpublishing | You can use the UpdateWorkbenchEventResult API to call back the event processing results to DataWorks. | ||
Pre-event for node freezing | ||||
Pre-event for node unfreezing | ||||
Data backfill operation event | Pre-event for data backfill | |||
Instance change events | Pre-event for instance freezing | |||
Pre-event for instance unfreezing | ||||
Pre-event for instance termination | ||||
Pre-event for instance rerunning | ||||
Pre-event for setting instance status to success | ||||
Approval Center events | Pre-event for permission request creation | You can use the CallbackExtension API to call back the event processing results to DataWorks. | ||
Data Quality verification events | Pre-event for batch creation of Data Quality rules | |||
Pre-event for batch update of Data Quality rules | ||||
Pre-event for Data Quality rule update | ||||
Pre-event for batch deletion of Data Quality rules | ||||
Pre-event for Data Quality monitoring creation | ||||
Pre-event for Data Quality monitoring update | ||||
Pre-event for Data Quality monitoring cloning | ||||
Pre-event for Data Quality monitoring alert subscription creation | ||||
Pre-event for batch deletion of Data Quality monitoring | ||||
Pre-event for Data Quality monitoring alert subscription update | ||||
Pre-event for Data Quality monitoring alert subscription deletion | ||||
Tenant level | Workspace change events | Pre-event for workspace deletion | ||
Data download events | Pre-event for data download (file generation) | |||
Pre-event for data download (file download) | ||||
Pre-event for data upload |
Appendix: Comparison of two deployment methods
Self-built service | Function Compute | |
Operation difficulty | The procedure is complex, involves deployment of servers and applications, and is prone to network and O&M issues. | The procedure is simple. You can develop and deploy an extension by using a single function. |
Cost | - | This method incurs Function Compute fees. For more information, see Billing overview. |
Supported events | This method supports various extension events. For more information about extension events, see Development reference: Event list and message format. | The Function Compute-based deployment method supports only the pre-event for data download, pre-event for data upload, pre-event for asset publishing, and pre-event for asset unpublishing. |