DataWorks:Develop and deploy an extension based on Function Compute

Permission description

• The Open Platform administrator and tenant administrator have read and write permissions on the developer backend. Other tenant-level roles and workspace-level roles have read-only permissions on the developer backend. For more information, see Manage permissions on global-level services.

• Only Alibaba Cloud accounts or RAM users to which the AliyunDataWorksFullAccess policy is attached can be used to configure settings on pages in the DataWorks console. For more information, see Manage permissions on the DataWorks services and the entities in the DataWorks console by using RAM policies.

Limits

• Only DataWorks Enterprise Edition or more advanced editions support the Extensions module.

  Note

  If DataWorks Enterprise Edition expires, extensions become invalid and cannot be triggered to check extension point events. If an extension is triggered to check an event and has not completed the check when DataWorks Enterprise Edition expires, the check is terminated and the result Check Passed is returned.

• You can associate multiple extensions with the same extension point event. This way, the associated extensions are triggered if the event occurs.

• Extensions that are deployed based on Function Compute can process only a pre-event for data download.

• If node groups, such as Platform for AI nodes, do-while nodes, and for-each nodes, trigger extension checks, you must wait for all inner nodes of the node groups to pass the checks before you can perform subsequent operations.

Billing

You are charged for using Function Compute. For more information, see Billing overview.

Step 1: Develop an extension on your on-premises machine

Before you deploy an extension based on Function Compute, you must develop the extension on your on-premises machine. Use the fc-java-core library to run a handler. You can download fc_test.zip to obtain the sample code. For more information, see Handler.

• For information about the formats of event messages sent by DataWorks, see Appendix: Message formats. You can configure the messageId parameter to obtain event details.

  Note

  Extensions that are deployed based on Function Compute can process only a pre-event for data download.

• Before an extension sends the processing result of an extension point event to DataWorks, you must check whether the extension can run as expected. An extension can return the following check result values:

  • OK: An extension point event passes the check of an extension.

  • FAIL: An extension point event fails the check of an extension. You must check and handle the reported error at the earliest opportunity to ensure that your program is run as expected.

  • WARN: An extension point event passes the check of an extension but a warning is reported for the event.

• When you develop the code of an extension, you can use parameters to improve code development and application efficiency. For example, you can use the built-in parameter extension.project.disabled of an extension to prevent the extension from taking effect for the specified workspace. For more information about the built-in parameters that you can use to develop extension code, see Advanced feature: Configure extension parameters.

Step 2: Deploy the extension in Function Compute

1. Log on to the Function Compute console.

2. Create a Function Compute service and the required function, and deploy the extension. The extension you developed will directly send specific event messages to the created service. For more information, see Create a service and Create a function.

   The following figure shows how to configure a function.

   

   • Select a runtime environment based on your code. If you use the sample package fc_test.zip provided in Step 1, set the Runtime parameter to Java 8. You must download the fc_test.zip package to your on-premises machine and upload it to the code package. You can configure the parameters based on your business requirements.

   • For more information about operations that you can perform on a function, see Manage functions.

Step 3: Register the extension in DataWorks

After you deploy the extension in Function Compute, you must register the extension in DataWorks. You can perform the following steps to register the extension in DataWorks.

1. Go to Open Platform.

2. In the left-side navigation pane of the Developer Backend tab, click Extensions.

3. Register the extension.

   Click Register Extension in the List of Extensions section. In the Select Deployment Method step of the Register Extension wizard, set the Please select your extension deployment type parameter to Deploy through function compute (Recommended) and then configure the parameters in the Register Extension step.

   

   The following table describes the parameters for registering an extension. You can configure the parameters based on your business requirements.

   Parameter	Description
   Extension Name	The name of the custom extension, which is used to identify the extension.
   Function Compute Service and Function	The Function Compute service and function on which you want to deploy the extension. The extension you developed will directly send specific event messages to the service.
   Processed Extension Points	Only the message that is triggered by a pre-event for data download can be processed. Note After you configure this parameter, the system automatically configures the Event and Applicable Service parameters.
   Owner	The extension owner. Users of the extension can contact the extension owner at the earliest opportunity when they encounter problems.
   Workspace for Testing	The workspace that is used to test the extension. To check whether the extension is effective, you do not need to publish the extension because the test workspace supports an end-to-end test that allows you to check whether the extension works as expected. In the test workspace, developers can trigger events to check whether DataWorks sends related messages to the extension by using EventBridge and whether the extension checks the events and sends the check results to DataWorks. Note If the extension point event that you want to process is a tenant-level extension point event, you do not need to configure the Workspace for Testing parameter.
   Extension Details Address	The URL of the extension details page. After you develop and deploy the extension, you can develop a web page to display how the extension works. Set this parameter to the page URL so that users can visit this web page to better understand and use the extension. For example, users can visit the web page to view the reason why a process is checked and blocked by the extension.
   Extension Documentation Address	The URL of the extension documentation page. After you develop and deploy the extension, you can develop a help documentation page. Set this parameter to the page URL so that users can know the business logic and properties of the extension.
   Parameters for Extension	The extension parameters that you want to use to improve the extension development and application efficiency. You can enter both the built-in parameters for typical scenarios and custom parameters for future reference. You can enter multiple parameters in the format of key=value. Make sure that each parameter occupies a separate line. For more information about how to use these parameters, see Advanced feature: Configure extension parameters.
   Options for Extension	The configuration items for the extension. The configuration items are provided for users to implement custom process management in different workspaces based on their business requirements. The extension developer must define each configuration item in a JSON string in the Register Extension dialog box. For example, the extension developer can allow users to manage the length of an SQL statement based on this parameter. For more information about the JSON format, see Advanced feature: Define options for an extension.

4. Click OK.

   After you register the extension, click Extensions in the left-side navigation pane. In the Extensions List section, you can perform the following operations on the extension:

   • View the extension: You can view the information such as the extension point, applicable service, extension code, and status of the extension.

   • Manage the extension: You can perform the operations such as submitting, publishing, or modifying the extension.

Sample code used to develop a custom extension

After you understand the precautions that you must take during the extension development procedure, you can develop extension code based on your business requirements. The following topics also provide examples on extension registration, development, and application in typical scenarios.

• Best practices for configuring the settings in DataWorks Open Platform to perform a lockdown

• Best practices for prohibiting the use of the MAX_PT function (advanced feature)

References

• For more information about Function Compute, see What is Function Compute?

• You can also deploy an extension based on a self-managed service. For more information, see Develop and deploy an extension based on a self-managed service.