DataWorks provides end-to-end data development and governance capabilities and a unified data development and governance process. This allows you to manage key stages in a process based on your business requirements. This topic describes the process management capabilities supported by DataWorks during data development.
Background information
DataWorks provides workspaces in standard mode and workspaces in basic mode. The node development process varies based on the workspace mode. For more information about DataWorks workspaces, see Comparison of basic mode and standard mode.
Node development process in a workspace in standard mode
Node development process in a workspace in basic mode
As shown in the preceding figures, you can manage key stages in a common node development process. For example, you can perform a check before you debug a node, or before you deploy a node to the development or production environment.
Stage | Sample check |
Before a node is run |
|
Before a node is deployed to the development environment |
|
Before a node is deployed to the production environment |
|
You can use DataWorks services, such as Open Platform and Data Asset Governance, to manage key stages in the data development process.
DataWorks service | Perform a check before a node is run | Perform a check before a node is deployed to the development environment | Perform a check before a node is deployed to the production environment | Description |
Data Asset Governance |
|
|
| Data Asset Governance in DataWorks provides multiple built-in check items. You can enable a check item based on your business requirements. This way, when you perform the related operation, the corresponding built-in check item is triggered to check the operation. You can proceed to the subsequent operations in the process only after the check is complete. |
Open Platform |
|
|
| If the built-in check items cannot meet your process management requirements, you can use Open Platform to register and develop programs as DataWorks extensions to check related events and add the check events to the overall data development process. |
The following sections use the data development process in a workspace in standard mode as an example to describe the process management capabilities.
Enable the built-in check items provided by Data Asset Governance
Data Asset Governance in DataWorks provides multiple built-in check items. You can enable a check item based on your business requirements. This way, when you perform the related operation, the corresponding built-in check item is triggered to check the operation. You can proceed to the subsequent operations in the process only after the check is complete.
Before a node is debugged, you can select Pre-event for Code Running for the Effective Checkpoint parameter based on your business requirements.
Before a node is deployed to the development environment, you can select Pre-event for Node Commit for the Effective Checkpoint parameter based on your business requirements.
Before a node is deployed to the production environment, you can select Pre-event for Node Deployment for the Effective Checkpoint parameter based on your business requirements.
Entry point for configuring check items and guidance
You need to enable check items in Data Asset Governance and specify the workspace in which the check items that you enabled take effect. For more information, see Configure governance items.
Develop custom extensions in Open Platform
If the built-in check items cannot meet your process management requirements, you can use Open Platform to register and develop programs as DataWorks extensions to check related events and add the check events to the overall data development process. The following content describes how a custom extension is used to check the node debugging.
DataWorks Open Platform provides the following modules: OpenAPI, OpenEvent, and Extensions. You can use the OpenEvent module to subscribe to event messages generated for the operations that you perform on the Data Studio page in the DataWorks console, use the Extensions module to create an extension to process the event messages, and use the OpenAPI module to send processing results to DataWorks. For information about the OpenEvent and Extensions modules, see Overview of OpenEvent and Overview of Extensions.
If you use the features provided by Open Platform to subscribe to an event message generated for a specific operation that you perform on the Data Studio page and create an extension to process the event message, a check is triggered when the operation is performed. The following figure shows a check process before node running.
Entry point for configuring check items and guidance
In Open Platform, you need to subscribe to event messages generated for the operations that you perform on the Data Studio page, develop an extension that can be used to process the event messages, publish the extension to DataWorks, and then specify the workspace in which the extension is enabled.
Before a node is debugged, you can subscribe to events related to node running, such as the pre-event for node running.
Before a node is deployed to the development environment, you can subscribe to events related to node committing, such as the pre-event for node committing and pre-event for table committing.
Before a node is deployed to the production environment, you can subscribe to events related to node deployment, such as the pre-event for node deployment and pre-event for table deployment.
For more information about the event types supported by Open Platform, see Overview of Extensions.
For more information about common operations that are supported in Open Platform, see Develop and deploy an extension based on a self-managed service.
For more information about the best practices in typical process management scenarios, see the following topics: