All Products
Search
Document Center

DataWorks:Best practices for prohibiting the use of the MAX_PT function (advanced feature)

Last Updated:Mar 31, 2025

DataWorks provides built-in checks, such as code review before node deployment and checks in Data Governance Center. DataWorks also supports custom checks. You can develop a custom check program based on your business requirements and connect the custom check program to DataWorks to manage the processes of nodes. This topic describes how to use an extension to check for a specified function that is prohibited to be used in a workspace. In this topic, an extension is used to check whether the code of a node contains the MAX_PT function when you commit and deploy the node.

Scenarios

This topic provides an example on how to use an extension to check whether the code of a node contains the MAX_PT function when you commit and deploy the node. The following table describes the check procedure.

No.

Core process

Core functionality

1

DataWorks uses the OpenEvent module to send the node committing and deployment event messages to EventBridge. The messages are generated when you commit and deploy a node in a workspace. Then, EventBridge filters event messages based on event rules and sends the event messages to your service.

When you enable event message subscription, you can specify the node committing and deployment events as the event types by setting the type parameter to dataworks:FileChange:CommitFile and dataworks:FileChange:DeployFile.

For more information about the configuration, see Best practices for prohibiting the use of the MAX_PT function (advanced feature).

2

Your offline or online service receives the event messages. You can configure your extension to achieve the following results:

  • For the event message that is generated when you commit a node in a specified workspace, the check for the event is not triggered and the event directly passes the check process.

  • For the event message that is generated when you deploy a node in a specified workspace, the check for the event is triggered and your extension checks whether the code of the node contains the MAX_PT function and responds to the event.

    If the MAX_PT function exists, DataWorks blocks the node committing or deployment process based on the event processing result that is returned by an API operation, or DataWorks sends you only an alert notification.

  • When you register the extension, you can configure the Parameters for Extension parameter to specify the workspace and event type for which the extension does not take effect. In this example, the extension does not take effect for node committing events. Therefore, the check for node committing events is not triggered.

  • When you register the extension, you can configure the Options for Extension parameter to specify the response to the event that does not meet the check conditions of the extension. In this example, you can select the response methods Alert and Disable.

For more information about the configuration, see Best practices for prohibiting the use of the MAX_PT function (advanced feature).

Prerequisites

  • EventBridge is activated. For more information, see Billing.

  • DataWorks is activated. For more information, see Purchase guide.

  • A workspace is created in DataWorks. For more information, see Create a workspace.

Procedure

Step 1: Configure a custom bus

  1. Log on to the EventBridge console. In the left-side navigation pane, click Event Buses.

  2. On the page that appears, click Quickly Create to Create Custom Event Bus.

    1. In the Event Bus step of the Create Custom Event Bus panel, configure the Name parameter and click Next Step.

    2. Click Skip for the Event Source, Event Rule, and Event Target steps.image

  3. In the left-side navigation pane, click Event Buses. On the Event Buses page, find the event bus that you created and click its name. The Overview page of the event bus appears.

    • In the left-side navigation pane, click Event Rules. On the page that appears, click Create Rule to create an event rule.

    • In this example, the custom bus is configured to receive node committing event messages and node deployment event messages. The following content provides an example on how to configure a demo and the core parameters for the event rule:

      1. Configure Basic Info: Enter a custom name for the event rule.

      2. Configure Event Pattern: In this step, configure the following parameters:

        • Event Source Type: Set it to Custom Event Source.

        • Event Source: Leave this parameter empty.

        • Pattern Content: Enter the following content in the JSON format.

          {
              "source": [
                  "acs.dataworks"
              ],
              "type": [
                  "dataworks:FileChange:CommitFile",
                  "dataworks:FileChange:DeployFile"
              ]
          }
          • source: the identifier of the service in which an event occurs. Set this parameter to acs.dataworks.

          • type: the type of the event that occurs in the service. Set this parameter to dataworks:FileChange:CommitFile and dataworks:FileChange:DeployFile.

        • Event Pattern Debugging: Change the values of the source and type parameters in this section and then click Test. If the test is successful, click Next Step.测试

      3. Configure Targets: In this step, configure the following parameters:

        • Service Type: Select HTTPS or HTTP. For more information about service types, see Manage event rules.

        • URL: Enter the URL for receiving messages pushed by the custom bus, such as https://Server address:Port number/extensions/consumer.

        • Body: Set it to Complete Event.

        • Network Type: Set it to Internet.事件目标

Step 2: Configure an event distribution channel

  1. Go to the Developer Backend tab.

    Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose More > Open Platform. On the page that appears, click Go to Open Platform. The Developer Backend tab appears.

  2. In the left-side navigation pane of the Developer Backend page, click OpenEvent. On the page that appears, click Add Event Distribution Channel. In the Add Event Distribution Channel dialog box, configure the parameters.

    • Workspace for Distribution of Event Messages: Select a workspace.

    • Specify Custom Event Bus in EventBridge for Distribution of Event Messages: Select the event bus that you created in Step 1.

  3. After you save the event distribution channel, find the event distribution channel on the OpenEvent page and click Enable in the Actions column.image

Step 3: Register and configure an extension

  1. Go to the Developer Backend tab.

    Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose More > Open Platform. On the page that appears, click Go to Open Platform. The Developer Backend tab appears.

  2. In the left-side navigation pane of the Developer Backend page, click Extensions. On the page that appears, click Register Extension. In the Register Extension dialog box, configure the parameters.

    1. Select Deployment Method: In this step, set Select a deployment method for your extension to Deploy Based on Self-managed Service.

    2. Register Extension: In this step, configure the following parameters:

      • Extension Name: Enter a custom name for the extension.

      • Processed Extension Points: Select Pre-event for Node Commit and Pre-event for Node Deployment.

      • Workspace for Testing: Specify a test workspace in which you can test the extension before you submit it.

      • Parameters for Extension

        You can use this parameter to specify the effective scope of the extension. For example, you can achieve the following results in a specified workspace:

        • The check for the node committing event is not triggered and the event directly passes the check process of the extension when you commit the node.

        • The check for the node deployment event is triggered when you deploy the node. If the event does not meet the check conditions of the extension, the event blocks the deployment process.

        Set this parameter to extension.project.commit-file.disabled=YourProjectId. Replace YourProjectId with the ID of the workspace in which the extension point event occurs. The extension you configure does not take effect for the extension point event.

      • Options for Extension: the response to the event. The following sample code shows the response methods that are used when the event does not meet the check conditions of the extension. The response methods include Alert and Disable.

        {
          "type":"object",
          "properties":{
            "checkStatus":{
              "type":"number",
              "title":"Check methods for the MAX-PT function",
              "x-decorator":"FormItem",
              "x-component":"Radio.Group",
              "x-decorator-props":{
                "tooltip":"Description file"
              },
              "x-component-props":{
                "dataSource":[
                  {
                    "value":"WARN",
                    "label":"Alert"
                  },
                  {
                    "value":"FAIL",
                    "label":"Disable"
                  }
                ],
                "mode":"multiple"
              }
            }
          }
        }
  3. After you complete the preceding configurations, click OK to save the registered extension.

  4. On the Extensions page, find the extension that you registered and click Submit in the Actions column.

    Note
    • Extensions are reviewed by DataWorks. In most cases, an extension review can be completed within three business days after you submit the extension for review.

    • To test the created extension, you must configure the Workspace for Testing parameter.

  5. After the extension passes the review, click Publish in the Actions column to publish and use the extension.

  6. Click Manage Extensions. You are navigated to the Extension page in SettingCenter. On the Extension page, find the created extension, enable the extension in the Enable column, and then click Configure in the Actions column to specify the response method for the event message that is generated when you commit a node in a workspace. The code of the node contains the MAX_PT function.image

Develop and configure the extension

EventBridge obtains JSON-formatted event messages sent by DataWorks based on an HTTP request, parses the event messages, and pushes the event messages to the destination service. The extension processes the event messages and returns the results to DataWorks.

Sample code

The sample code shows how to call the UpdateIDEEventResult operation with the messageId parameter configured to obtain the details of an event message, and then logically determine whether the specified function is contained. Then, OpenAPI allows you to call the UpdateIDEEventResult operation to send the processing result of the event to DataWorks. For more information, see Develop and deploy an extension based on a self-managed service.

Environment requirement: Java 8 and Maven. Maven is a build automation tool for Java.

package com.aliyun.dataworks.demo;

import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.JSONObject;
import com.aliyun.dataworks.config.Constants;
import com.aliyun.dataworks.config.EventCheckEnum;
import com.aliyun.dataworks.config.ExtensionParamProperties;
import com.aliyun.dataworks.services.DataWorksOpenApiClient;
import com.aliyuncs.IAcsClient;
import com.aliyuncs.dataworks_public.model.v20200518.*;
import com.aliyuncs.exceptions.ClientException;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.*;

/**
 * @author dataworks demo
 */
@RestController
@RequestMapping("/extensions")
public class ExtensionsController {

    @Autowired(required = false)
    private DataWorksOpenApiClient dataWorksOpenApiClient;

    @Autowired
    private ExtensionParamProperties extensionParamProperties;

    /**
     * Receive event messages that are sent from EventBridge.
     * @param jsonParam
     */
    @PostMapping("/consumer")
    public void consumerEventBridge(@RequestBody String jsonParam){
        JSONObject jsonObj = JSON.parseObject(jsonParam);
        String eventCode = jsonObj.getString(Constants.EVENT_CODE_FILED);
        if(Constants.COMMIT_FILE_EVENT_CODE.equals(eventCode) || Constants.DEPLOY_FILE_EVENT_CODE.equals(eventCode)){
            // Initialize the client.
            IAcsClient client = dataWorksOpenApiClient.createClient();
            try {
                // The information about the parameters for the current event.
                String messageId = jsonObj.getString("id");
                JSONObject data = jsonObj.getObject("data", JSONObject.class);
                Long projectId = data.getLong("projectId");

                // Initialize the event callback.
                UpdateIDEEventResultRequest updateIDEEventResultRequest = new UpdateIDEEventResultRequest();
                updateIDEEventResultRequest.setMessageId(messageId);
                updateIDEEventResultRequest.setExtensionCode(extensionParamProperties.getExtensionCode());

                // Query the data snapshot that is generated when the extension point event is triggered.
                GetIDEEventDetailRequest getIDEEventDetailRequest = new GetIDEEventDetailRequest();
                getIDEEventDetailRequest.setMessageId(messageId);
                getIDEEventDetailRequest.setProjectId(projectId);
                GetIDEEventDetailResponse getIDEEventDetailResponse = client.getAcsResponse(getIDEEventDetailRequest);
                String content = getIDEEventDetailResponse.getEventDetail().getCommittedFile().getContent();

                // Check whether the code contains the function that is prohibited to be used.
                if(content.contains(Constants.CHECK_CODE)){
                    // Query the option settings of the extension in a workspace.
                    GetOptionValueForProjectRequest getOptionValueForProjectRequest = new GetOptionValueForProjectRequest();
                    getOptionValueForProjectRequest.setProjectId(String.valueOf(projectId));
                    getOptionValueForProjectRequest.setExtensionCode(extensionParamProperties.getExtensionCode());
                    GetOptionValueForProjectResponse getOptionValueForProjectResponse = client.getAcsResponse(getOptionValueForProjectRequest);
                    JSONObject jsonObject = JSON.parseObject(getOptionValueForProjectResponse.getOptionValue());
                    // Note: You must specify the format based on the value of the Options for Extension parameter in DataWorks.
                    String checkStatus = jsonObject.getString("checkStatus");
                    updateIDEEventResultRequest.setCheckResult(checkStatus);
                    updateIDEEventResultRequest.setCheckResultTip("The node code contains the specific function that is prohibited to be used.");
                }else{// The callback is successful.
                    updateIDEEventResultRequest.setCheckResult(EventCheckEnum.OK.getCode());
                    updateIDEEventResultRequest.setCheckResultTip(EventCheckEnum.OK.getName());
                }
                // The extension processes the event messages and calls an API operation of DataWorks to send the processing result to DataWorks.
                UpdateIDEEventResultResponse acsResponse = client.getAcsResponse(updateIDEEventResultRequest);
                // The ID of the request. You can troubleshoot errors based on the ID.
                System.out.println("acsResponse:" + acsResponse.getRequestId());
            } catch (ClientException e) {
                // The ID of the request. You can troubleshoot errors based on the ID.
                System.out.println("RequestId:" + e.getRequestId());
                // The status code of an error.
                System.out.println("ErrCode:" + e.getErrCode());
                // The description of an error.
                System.out.println("ErrMsg:" + e.getErrMsg());
            }
        }else{
            System.out.println("Failed to filter out other types of events. Check the parameter configurations.");
        }
    }
}

Sample project deployment

  1. Prepare the environment and the project.

  2. Select a deployment mode.

    • On-premises deployment: After you package the project file into a JAR package, run the java -jar yourapp.jar command on the on-premises server in which Java 8 and Maven are deployed to start the service program.

    • Cloud-based deployment: After you package the project file into a JAR package, upload the package to the related runtime environment, such as Docker containers and cloud servers, for deployment.

    Note

    After you deploy the service program, make sure that EventBridge can access the service over the Internet.

  3. After you download the project file, enter the project root directory and run the packaging command to package the project file into a JAR package.

    mvn clean package -Dmaven.test.skip=true spring-boot:repackage
  4. Execute the JAR package:

    java -jar target/extensions-demo-maxpt-1.0.jar

    The project is successfully started as shown in the following figure.maxptEnter http://localhost:8080/index in the address bar of a browser and press Enter. If "hello world!" is returned, the extension is successfully deployed. You can subscribe to event messages after network connections are established between DataWorks and your extension and between EventBridge and your extension.

Verify results

After you deploy the code and establish a network connection to EventBridge, you can verify the code in the workspace where the extension is enabled.

Verification steps

  1. On the Data Studio page, create a node, edit the prohibited MAX_PT function in the node, save the modification, and then commit the node.

  2. Click Deploy to go to the Deployment Package Creation page and deploy the node directly. Verification of the extension is immediately triggered.

    Note

    Based on the configurations, the extension does not take effect when you commit a node in the specified workspace. That means when you commit a node in which the MAX_PT function is contained, the extension is not triggered. However, when you deploy a node in which the MAX_PT function is contained, the extension is triggered.