DataWorks:Best practices for prohibiting the use of the MAX_PT function (advanced feature)

Step 1: Configure a custom bus

1. Log on to the EventBridge console. In the left-side navigation pane, click Event Buses.

2. On the page that appears, click Quickly Create to Create Custom Event Bus.

   1. In the Event Bus step of the Create Custom Event Bus panel, configure the Name parameter and click Next Step.

   2. Click Skip for the Event Source, Event Rule, and Event Target steps.

3. In the left-side navigation pane, click Event Buses. On the Event Buses page, find the event bus that you created and click its name. The Overview page of the event bus appears.

   • In the left-side navigation pane, click Event Rules. On the page that appears, click Create Rule to create an event rule.

   • In this example, the custom bus is configured to receive node committing event messages and node deployment event messages. The following content provides an example on how to configure a demo and the core parameters for the event rule:

     1. Configure Basic Info: Enter a custom name for the event rule.

     2. Configure Event Pattern: In this step, configure the following parameters:

        • Event Source Type: Set it to Custom Event Source.

        • Event Source: Leave this parameter empty.

        • Pattern Content: Enter the following content in the JSON format.

          {
              "source": [
                  "acs.dataworks"
              ],
              "type": [
                  "dataworks:FileChange:CommitFile",
                  "dataworks:FileChange:DeployFile"
              ]
          }

          • source: the identifier of the service in which an event occurs. Set this parameter to acs.dataworks.

          • type: the type of the event that occurs in the service. Set this parameter to dataworks:FileChange:CommitFile and dataworks:FileChange:DeployFile.

        • Event Pattern Debugging: Change the values of the source and type parameters in this section and then click Test. If the test is successful, click Next Step.

     3. Configure Targets: In this step, configure the following parameters:

        • Service Type: Select HTTPS or HTTP. For more information about service types, see Manage event rules.

        • URL: Enter the URL for receiving messages pushed by the custom bus, such as https://Server address:Port number/extensions/consumer.

        • Body: Set it to Complete Event.

        • Network Type: Set it to Internet.

Step 2: Configure an event distribution channel

1. Go to the Developer Backend tab.

   Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose More > Open Platform. On the page that appears, click Go to Open Platform. The Developer Backend tab appears.

2. In the left-side navigation pane of the Developer Backend page, click OpenEvent. On the page that appears, click Add Event Distribution Channel. In the Add Event Distribution Channel dialog box, configure the parameters.

   • Workspace for Distribution of Event Messages: Select a workspace.

   • Specify Custom Event Bus in EventBridge for Distribution of Event Messages: Select the event bus that you created in Step 1.

3. After you save the event distribution channel, find the event distribution channel on the OpenEvent page and click Enable in the Actions column.

Step 3: Register and configure an extension

1. Go to the Developer Backend tab.

   Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose More > Open Platform. On the page that appears, click Go to Open Platform. The Developer Backend tab appears.

2. In the left-side navigation pane of the Developer Backend page, click Extensions. On the page that appears, click Register Extension. In the Register Extension dialog box, configure the parameters.

   1. Select Deployment Method: In this step, set Select a deployment method for your extension to Deploy Based on Self-managed Service.

   2. Register Extension: In this step, configure the following parameters:

      • Extension Name: Enter a custom name for the extension.

      • Processed Extension Points: Select Pre-event for Node Commit and Pre-event for Node Deployment.

      • Workspace for Testing: Specify a test workspace in which you can test the extension before you submit it.

      • Parameters for Extension

        You can use this parameter to specify the effective scope of the extension. For example, you can achieve the following results in a specified workspace:

        • The check for the node committing event is not triggered and the event directly passes the check process of the extension when you commit the node.

        • The check for the node deployment event is triggered when you deploy the node. If the event does not meet the check conditions of the extension, the event blocks the deployment process.

        Set this parameter to extension.project.commit-file.disabled=YourProjectId. Replace YourProjectId with the ID of the workspace in which the extension point event occurs. The extension you configure does not take effect for the extension point event.

      • Options for Extension: the response to the event. The following sample code shows the response methods that are used when the event does not meet the check conditions of the extension. The response methods include Alert and Disable.

        {
          "type":"object",
          "properties":{
            "checkStatus":{
              "type":"number",
              "title":"Check methods for the MAX-PT function",
              "x-decorator":"FormItem",
              "x-component":"Radio.Group",
              "x-decorator-props":{
                "tooltip":"Description file"
              },
              "x-component-props":{
                "dataSource":[
                  {
                    "value":"WARN",
                    "label":"Alert"
                  },
                  {
                    "value":"FAIL",
                    "label":"Disable"
                  }
                ],
                "mode":"multiple"
              }
            }
          }
        }

3. After you complete the preceding configurations, click OK to save the registered extension.

4. On the Extensions page, find the extension that you registered and click Submit in the Actions column.

   Note

   • Extensions are reviewed by DataWorks. In most cases, an extension review can be completed within three business days after you submit the extension for review.

   • To test the created extension, you must configure the Workspace for Testing parameter.

5. After the extension passes the review, click Publish in the Actions column to publish and use the extension.

6. Click Manage Extensions. You are navigated to the Extension page in SettingCenter. On the Extension page, find the created extension, enable the extension in the Enable column, and then click Configure in the Actions column to specify the response method for the event message that is generated when you commit a node in a workspace. The code of the node contains the MAX_PT function.

Sample code

The sample code shows how to call the UpdateIDEEventResult operation with the messageId parameter configured to obtain the details of an event message, and then logically determine whether the specified function is contained. Then, OpenAPI allows you to call the UpdateIDEEventResult operation to send the processing result of the event to DataWorks. For more information, see Develop and deploy an extension based on a self-managed service.

Environment requirement: Java 8 and Maven. Maven is a build automation tool for Java.

package com.aliyun.dataworks.demo;

import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.JSONObject;
import com.aliyun.dataworks.config.Constants;
import com.aliyun.dataworks.config.EventCheckEnum;
import com.aliyun.dataworks.config.ExtensionParamProperties;
import com.aliyun.dataworks.services.DataWorksOpenApiClient;
import com.aliyuncs.IAcsClient;
import com.aliyuncs.dataworks_public.model.v20200518.*;
import com.aliyuncs.exceptions.ClientException;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.*;

/**
 * @author dataworks demo
 */
@RestController
@RequestMapping("/extensions")
public class ExtensionsController {

    @Autowired(required = false)
    private DataWorksOpenApiClient dataWorksOpenApiClient;

    @Autowired
    private ExtensionParamProperties extensionParamProperties;

    /**
     * Receive event messages that are sent from EventBridge.
     * @param jsonParam
     */
    @PostMapping("/consumer")
    public void consumerEventBridge(@RequestBody String jsonParam){
        JSONObject jsonObj = JSON.parseObject(jsonParam);
        String eventCode = jsonObj.getString(Constants.EVENT_CODE_FILED);
        if(Constants.COMMIT_FILE_EVENT_CODE.equals(eventCode) || Constants.DEPLOY_FILE_EVENT_CODE.equals(eventCode)){
            // Initialize the client.
            IAcsClient client = dataWorksOpenApiClient.createClient();
            try {
                // The information about the parameters for the current event.
                String messageId = jsonObj.getString("id");
                JSONObject data = jsonObj.getObject("data", JSONObject.class);
                Long projectId = data.getLong("projectId");

                // Initialize the event callback.
                UpdateIDEEventResultRequest updateIDEEventResultRequest = new UpdateIDEEventResultRequest();
                updateIDEEventResultRequest.setMessageId(messageId);
                updateIDEEventResultRequest.setExtensionCode(extensionParamProperties.getExtensionCode());

                // Query the data snapshot that is generated when the extension point event is triggered.
                GetIDEEventDetailRequest getIDEEventDetailRequest = new GetIDEEventDetailRequest();
                getIDEEventDetailRequest.setMessageId(messageId);
                getIDEEventDetailRequest.setProjectId(projectId);
                GetIDEEventDetailResponse getIDEEventDetailResponse = client.getAcsResponse(getIDEEventDetailRequest);
                String content = getIDEEventDetailResponse.getEventDetail().getCommittedFile().getContent();

                // Check whether the code contains the function that is prohibited to be used.
                if(content.contains(Constants.CHECK_CODE)){
                    // Query the option settings of the extension in a workspace.
                    GetOptionValueForProjectRequest getOptionValueForProjectRequest = new GetOptionValueForProjectRequest();
                    getOptionValueForProjectRequest.setProjectId(String.valueOf(projectId));
                    getOptionValueForProjectRequest.setExtensionCode(extensionParamProperties.getExtensionCode());
                    GetOptionValueForProjectResponse getOptionValueForProjectResponse = client.getAcsResponse(getOptionValueForProjectRequest);
                    JSONObject jsonObject = JSON.parseObject(getOptionValueForProjectResponse.getOptionValue());
                    // Note: You must specify the format based on the value of the Options for Extension parameter in DataWorks.
                    String checkStatus = jsonObject.getString("checkStatus");
                    updateIDEEventResultRequest.setCheckResult(checkStatus);
                    updateIDEEventResultRequest.setCheckResultTip("The node code contains the specific function that is prohibited to be used.");
                }else{// The callback is successful.
                    updateIDEEventResultRequest.setCheckResult(EventCheckEnum.OK.getCode());
                    updateIDEEventResultRequest.setCheckResultTip(EventCheckEnum.OK.getName());
                }
                // The extension processes the event messages and calls an API operation of DataWorks to send the processing result to DataWorks.
                UpdateIDEEventResultResponse acsResponse = client.getAcsResponse(updateIDEEventResultRequest);
                // The ID of the request. You can troubleshoot errors based on the ID.
                System.out.println("acsResponse:" + acsResponse.getRequestId());
            } catch (ClientException e) {
                // The ID of the request. You can troubleshoot errors based on the ID.
                System.out.println("RequestId:" + e.getRequestId());
                // The status code of an error.
                System.out.println("ErrCode:" + e.getErrCode());
                // The description of an error.
                System.out.println("ErrMsg:" + e.getErrMsg());
            }
        }else{
            System.out.println("Failed to filter out other types of events. Check the parameter configurations.");
        }
    }
}

Verification steps

1. On the Data Studio page, create a node, edit the prohibited MAX_PT function in the node, save the modification, and then commit the node.

2. Click Deploy to go to the Deployment Package Creation page and deploy the node directly. Verification of the extension is immediately triggered.

   Note

   Based on the configurations, the extension does not take effect when you commit a node in the specified workspace. That means when you commit a node in which the MAX_PT function is contained, the extension is not triggered. However, when you deploy a node in which the MAX_PT function is contained, the extension is triggered.