DataWorks:Develop and deploy an extension based on a self-managed service

Permission description

• The Open Platform administrator and tenant administrator have read and write permissions on the developer backend. Other tenant-level roles and workspace-level roles have read-only permissions on the developer backend. For more information, see Manage permissions on global-level services.

• Only Alibaba Cloud accounts or RAM users to which the AliyunDataWorksFullAccess policy is attached can be used to configure settings on pages in the DataWorks console. For more information, see Manage permissions on the DataWorks services and the entities in the DataWorks console by using RAM policies.

Limits

• Only DataWorks Enterprise Edition or more advanced editions support the Extensions module.

  Note

  If DataWorks Enterprise Edition expires, extensions become invalid and cannot be triggered to check extension point events. If an extension is triggered to check an event and has not completed the check when DataWorks Enterprise Edition expires, the check is terminated and the result Check Passed is returned.

• You can associate multiple extensions with the same extension point event. This way, the associated extensions are triggered if the event occurs.

• Extension point events are classified into tenant-level events and workspace-level events. You can select one of the preceding types of events when you register an extension. For information about the types of extension point events that are supported by DataWorks, see Supported extension point events.

• If node groups, such as Platform for AI nodes, do-while nodes, and for-each nodes, trigger extension checks, you must wait for all inner nodes of the node groups to pass the checks before you can perform subsequent operations.

Process

You can develop and deploy an extension based on a self-managed service and establish a connection between the extension and DataWorks to implement custom process control by using the OpenEvent and OpenAPI modules of DataWorks Open Platform.

1. Use OpenEvent to subscribe to extension point events.

   The OpenEvent module allows you to subscribe to specific extension point events by using EventBridge. After you subscribe to the message of an event, you can obtain the details of the message. For example, if you click the Submit icon to commit a node, a file committing event is triggered, and you can receive a message that contains the information, such as node ID, operator UID, and workspace ID. For more information, see Step 1: Enable event message subscription.

2. Use Extensions to process extension point events.

   You can register your service program as an extension. After you receive the event message that you subscribe, use the extension to process the event. For more information, see Step 2: Register an extension.

3. Use OpenAPI to call an API operation to send the processing result of an extension point event to DataWorks.

   For more information, see Step 3: Develop an extension.

   When an extension point event is triggered and processed by an extension, the event enters the Checking state. After the event is checked, the extension calls an API operation to send the processing result to DataWorks for reference in the subsequent data processing. An extension can return the following check result values:

4. Deploy the extension as a local application. For more information, see Step 4: Deploy an extension.

Step 1: Enable event message subscription

The Extensions module works with the OpenEvent module to enable automatic response to extension point events in the following manner: DataWorks sends event messages to extensions by using the OpenEvent module. The extensions receive the messages and check the extension point events. Before you develop and use an extension, you must enable the event message subscription feature. You must also obtain the information about EventBridge and check the message bodies for different types of events to facilitate extension code development.

• For information about how to enable the event message subscription feature, see Enable event message subscription.

• For information about the message formats for different types of events that you subscribe, see Appendix: Message formats.

Step 2: Register an extension

Before you develop and use an extension, you must register the extension in DataWorks and obtain the extension code for later code development. To register an extension, perform the following steps:

1. Go to Open Platform.

2. In the left-side navigation pane of the Developer Backend tab, click Extensions.

3. Register the extension.

   Click Register Extension in the List of Extensions section. In the Select Deployment Method step of the Register Extension wizard, set the Please select your extension deployment type parameter to Deploy through self-built services and then configure the parameters in the Register Extension step.

   

   The following table describes the parameters for registering an extension. You can configure the parameters based on your business requirements.

   Parameter	Description
   Extension Name	The name of the custom extension, which is used to identify the extension.
   Processed Extension Points	The type of the extension point event that you want the extension to check. For more information about supported extension point events, see Supported extension point events. You can configure this parameter based on your business requirements. Note After you configure this parameter, the system automatically configures the Event and Applicable Service parameters. Extension point events are classified into tenant-level events and workspace-level events. You can select one of the preceding types of events when you register an extension. For information about the types of extension point events that are supported by DataWorks, see Supported extension point events. Extensions that are deployed based on Function Compute can process only a pre-event for data download.
   Owner	The extension owner. Users of the extension can contact the extension owner at the earliest opportunity when they encounter problems.
   Workspace for Testing	The workspace that is used to test the extension. To check whether the extension is effective, you do not need to publish the extension because the test workspace supports an end-to-end test that allows you to check whether the extension works as expected. In the test workspace, developers can trigger events to check whether DataWorks sends related messages to the extension by using EventBridge and whether the extension checks the events and sends the check results to DataWorks. Note If the extension point event that you want to process is a tenant-level extension point event, you do not need to configure the Workspace for Testing parameter.
   Extension Details Address	The URL of the extension details page. After you develop and deploy the extension, you can develop a web page to display how the extension works. Set this parameter to the page URL so that users can visit this web page to better understand and use the extension. For example, users can visit the web page to view the reason why a process is checked and blocked by the extension.
   Extension Documentation Address	The URL of the extension documentation page. After you develop and deploy the extension, you can develop a help documentation page. Set this parameter to the page URL so that users can know the business logic and properties of the extension.
   Parameters for Extension	The extension parameters that you want to use to improve the extension development and application efficiency. You can enter both the built-in parameters for typical scenarios and custom parameters for future reference. You can enter multiple parameters in the format of key=value. Make sure that each parameter occupies a separate line. For more information about how to use these parameters, see Advanced feature: Configure extension parameters.
   Options for Extension	The configuration items for the extension. The configuration items are provided for users to implement custom process management in different workspaces based on their business requirements. The extension developer must define each configuration item in a JSON string in the Register Extension dialog box. For example, the extension developer can allow users to manage the length of an SQL statement based on this parameter. For more information about the JSON format, see Advanced feature: Define options for an extension.

4. Click OK.

   After you register the extension, click Extensions in the left-side navigation pane. In the Extensions List section, you can perform the following operations on the extension:

   • View the extension: You can view the information such as the extension point, applicable service, extension code, and status of the extension.

   • Manage the extension: You can perform the operations such as submitting, publishing, or modifying the extension.

Step 3: Develop an extension

When you develop code for an extension, take note of the following items:

1. Subscribe to and parse the messages of extension point events.

   • The OpenEvent module of DataWorks Open Platform uses EventBridge to distribute event messages. You can subscribe to event messages by using EventBridge. For more information, see What is EventBridge?

   • For information about the message formats of extension point events that are sent to EventBridge, see Appendix: Message formats.

2. Obtain the snapshots of extension point events and process events based on the snapshots.

   • You can configure the messageId parameter in the API operation that you called to obtain the snapshots of extension point events.

     Note

     • You can call the UpdateIDEEventResult operation for extension point events in DataStudio and the UpdateWorkbenchEventResult operation for extension point events in Operation Center.

     • You can obtain the value of the messageId parameter by parsing event messages.

   • You can use parameters to improve code development and application efficiency of an extension. For example, you can use the built-in parameter extension.project.disabled of an extension to prevent the extension from taking effect for the specified workspace. For more information about the built-in parameters that you can use to develop extension code, see Advanced feature: Configure extension parameters.

3. Send the processing result of the extension to DataWorks.

   You can call an API operation to configure the extension to return the processing results of extension point events to DataWorks.

   Note

   You can call the UpdateIDEEventResult operation for extension point events in DataStudio and the UpdateWorkbenchEventResult operation for extension point events in Operation Center.

Step 4: Deploy an extension

After you develop and debug the extension code, you can deploy the extension as an application service on your on-premises machine or an Alibaba Cloud Elastic Compute Service (ECS) instance for subsequent use.

Sample code used to develop a custom extension

After you understand the precautions that you must take during the extension development procedure, you can develop extension code based on your business requirements. The following topics also provide examples on extension registration, development, and application in typical scenarios.

• Best practices for configuring the settings in DataWorks Open Platform to perform a lockdown

• Best practices for prohibiting the use of the MAX_PT function (advanced feature)

References

• For information about the message formats for different types of events, see Appendix: Message formats.

• The OpenEvent module allows you to subscribe to extension point events by using EventBridge. For more information, see Overview.

• For more information about the extension point events that can be processed by DataWorks extensions, see Overview.

• You can also deploy an extension based on Function Compute. For more information, see Develop and deploy a custom extension.