Best practices for prohibiting the use of the MAX_PT function (advanced feature) - DataWorks

DataWorks provides built-in checks, such as code review before node deployment and checks in Data Governance Center. DataWorks also supports custom checks. You can develop a custom check program based on your business requirements and connect the custom check program to DataWorks to manage the processes of nodes. This topic describes how to use an extension to check for a specified function that is prohibited to be used in a workspace. In this topic, an extension is used to check whether the code for a node contains the MAX_PT function when you commit and deploy the node.

Scenarios

This topic provides an example on how to use an extension to check whether the code for a node contains the MAX_PT function when you commit and deploy the node. The following table describes the check procedure.


No.	Core process	Core functionality
1	DataWorks uses the OpenEvent module to send the node commit and deployment event messages to EventBridge. The messages are generated when you commit and deploy a node in a workspace. Then, EventBridge filters event messages based on event rules and sends the event messages to your service.	When you enable event message subscription, you can specify the node commit and deployment events as the event types by setting the type parameter to dataworks:FileChange:CommitFile and dataworks:FileChange:DeployFile. For more information, see Enable and configure event message subscription (OpenEvent).
2	Your offline or online service receives the event messages. You can configure your extension to achieve the following results: For the event message that is generated when you commit a node in a specified workspace, the check for the event is not triggered and the event directly passes the check process. For the event message that is generated when you deploy a node in a specified workspace, the check for the event is triggered and your extension checks whether the code for the node contains the MAX_PT function and responds to the event. If the MAX_PT function exists, DataWorks blocks the node commit or deployment process based on the event processing result that is returned by an API operation, or DataWorks sends you only an alert notification.	When you register the extension, you can configure the Parameters for Extension parameter to specify the workspace and event type for which the extension does not take effect. In this example, the extension does not take effect for node commit events. Therefore, the check for node commit events is not triggered. When you register the extension, you can configure the Options for Extension parameter to specify the response to the event that does not meet the check conditions of the extension. In this example, you can select the response methods Alert and Disable. For more information, see Register and configure an extension (Extensions).

Enable and configure event message subscription (OpenEvent)

This section describes the core configuration steps and precautions for enabling and configuring event message subscription. For more information, see Enable event message subscription.

In the EventBridge console, create a custom bus. You do not need to configure the parameters in the Event Source, Event Rule, and Event Target steps.
Create an event rule for the custom bus in the EventBridge console.
In this example, the custom bus is configured to receive node commit event messages and node deployment event messages. The following content provides an example on how to configure a demo and the core parameters for the event rule:
1. Configure an event pattern.
```
{
    "source": [
        "acs.dataworks"
    ],
    "type": [
        "dataworks:FileChange:CommitFile",
        "dataworks:FileChange:DeployFile"
    ]
}
```
  - source: the identifier of the product in which an event occurs. Set this parameter to acs.dataworks.
  - type: the type of the event that occurs in the product. Set this parameter to dataworks:FileChange:CommitFile and dataworks:FileChange:DeployFile. You can change the values of the source and type parameters in the Event Pattern Debugging section and then click Test. If the test is successful, click Next Step.
2. In the Configure Targets step, set Service Type to HTTPS and enter a valid URL. Use the default settings for other parameters.
Go to the Open Platform page in the DataWorks console. In the left-side navigation tree, click OpenEvent. On the page that appears, add an event distribution channel.
On the Open Platform page in the DataWorks console, enable the added event distribution channel.

Register and configure an extension (Extensions)

This section describes the core configuration steps and precautions for registering and configuring an extension. For more information, see Preparations.

Go to the Open Platform page in the DataWorks console. In the left-side navigation tree, click Extensions. On the page that appears, click Register Extensions to register an extension.

This section provides an example on how to register and configure an extension to process the node commit event message that is generated when you commit a node on the DataStudio page. The following content provides the configurations of the core parameters. You can configure other parameters by following the on-screen instructions.


Core parameter	Description
Processed Extension Points	Select Pre-event for Node Commit and Pre-event for Node Deployment.
Parameters for Extension	You can use this parameter to specify the effective scope of the extension. For example, you can achieve the following results in a specified workspace: The check for the node commit event is not triggered and the event directly passes the check process of the extension when you commit the node. The check for the node deployment event is triggered when you deploy the node. If the event does not meet the check conditions of the extension, the event blocks the deployment process. Set this parameter to `extension.project.commit-file.disabled=YourProjectId`. Replace YourProjectId with the ID of the workspace in which the extension point event occurs. The extension you configure does not take effect for the extension point event.
Options for Extension	The response to the event. The following sample code shows the response methods that are used when the event does not meet the check conditions of the extension. The response methods include Alert and Disable. `{ "type":"object", "properties":{ "checkStatus":{ "type":"number", "title":"Check methods for the MAX-PT function", "x-decorator":"FormItem", "x-component":"Radio.Group", "x-decorator-props":{ "tooltip":"Description file" }, "x-component-props":{ "dataSource":[ { "value":"WARN", "label":"Alert" }, { "value":"FAIL", "label":"Disable" } ], "mode":"multiple" } } } }`

Develop and deploy the extension.

Configure DataWorks SDK for Java. For more information, see Install SDKs for Java. Sample code:

package com.aliyun.dataworks.demo;

import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.JSONObject;
import com.aliyun.dataworks.config.Constants;
import com.aliyun.dataworks.config.EventCheckEnum;
import com.aliyun.dataworks.config.ExtensionParamProperties;
import com.aliyun.dataworks.services.DataWorksOpenApiClient;
import com.aliyuncs.IAcsClient;
import com.aliyuncs.dataworks_public.model.v20200518.*;
import com.aliyuncs.exceptions.ClientException;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.*;

/**
 * @author dataworks demo
 */
@RestController
@RequestMapping("/extensions")
public class ExtensionsController {

    @Autowired(required = false)
    private DataWorksOpenApiClient dataWorksOpenApiClient;

    @Autowired
    private ExtensionParamProperties extensionParamProperties;

    /**
     * Receive event messages that are sent from EventBridge.
     * @param jsonParam
     */
    @PostMapping("/consumer")
    public void consumerEventBridge(@RequestBody String jsonParam){
        JSONObject jsonObj = JSON.parseObject(jsonParam);
        String eventCode = jsonObj.getString(Constants.EVENT_CODE_FILED);
        if(Constants.COMMIT_FILE_EVENT_CODE.equals(eventCode) || Constants.DEPLOY_FILE_EVENT_CODE.equals(eventCode)){
            // Initialize the client.
            IAcsClient client = dataWorksOpenApiClient.createClient();
            try {
                // The information about the parameters for the current event.
                String messageId = jsonObj.getString("id");
                JSONObject data = jsonObj.getObject("data", JSONObject.class);
                Long projectId = data.getLong("projectId");

                // Initialize the event callback.
                UpdateIDEEventResultRequest updateIDEEventResultRequest = new UpdateIDEEventResultRequest();
                updateIDEEventResultRequest.setMessageId(messageId);
                updateIDEEventResultRequest.setExtensionCode(extensionParamProperties.getExtensionCode());

                // Query the data snapshot that is generated when the extension point event is triggered.
                GetIDEEventDetailRequest getIDEEventDetailRequest = new GetIDEEventDetailRequest();
                getIDEEventDetailRequest.setMessageId(messageId);
                getIDEEventDetailRequest.setProjectId(projectId);
                GetIDEEventDetailResponse getIDEEventDetailResponse = client.getAcsResponse(getIDEEventDetailRequest);
                String content = getIDEEventDetailResponse.getEventDetail().getCommittedFile().getContent();

                // Check whether the code contains the function that is prohibited to be used.
                if(content.contains(Constants.CHECK_CODE)){
                    // Query the configuration item that is specified for the Options for Extension parameter in a workspace.
                    GetOptionValueForProjectRequest getOptionValueForProjectRequest = new GetOptionValueForProjectRequest();
                    getOptionValueForProjectRequest.setProjectId(String.valueOf(projectId));
                    getOptionValueForProjectRequest.setExtensionCode(extensionParamProperties.getExtensionCode());
                    GetOptionValueForProjectResponse getOptionValueForProjectResponse = client.getAcsResponse(getOptionValueForProjectRequest);
                    JSONObject jsonObject = JSON.parseObject(getOptionValueForProjectResponse.getOptionValue());
                    // Note: You must specify the format based on the value of the Options for Extension parameter in DataWorks.
                    String checkStatus = jsonObject.getString("checkStatus");
                    updateIDEEventResultRequest.setCheckResult(checkStatus);
                    updateIDEEventResultRequest.setCheckResultTip("The node code contains the specific function that is prohibited to be used.");
                }else{// The callback is successful.
                    updateIDEEventResultRequest.setCheckResult(EventCheckEnum.OK.getCode());
                    updateIDEEventResultRequest.setCheckResultTip(EventCheckEnum.OK.getName());
                }
                // The extension processes the event messages and calls an API operation of DataWorks to send the processing result to DataWorks.
                UpdateIDEEventResultResponse acsResponse = client.getAcsResponse(updateIDEEventResultRequest);
                // The ID of the request. You can troubleshoot errors based on the ID.
                System.out.println("acsResponse:" + acsResponse.getRequestId());
            } catch (ClientException e) {
                // The ID of the request. You can troubleshoot errors based on the ID.
                System.out.println("RequestId:" + e.getRequestId());
                // The status code of an error.
                System.out.println("ErrCode:" + e.getErrCode());
                // The description of an error.
                System.out.println("ErrMsg:" + e.getErrMsg());
            }
        }else{
            System.out.println("Failed to filter out other types of events. Check the parameter configurations.");
        }
    }
}

Enable the extension.
After the extension is registered, click Extensions management. In the left-side navigation pane of the SettingCenter page, click Extension. On the Extension page, turn on the switch for the registered extension in the Enable column and complete the authorization by following the on-screen instructions to enable the extension. Then, click Configure in the Actions column of the registered extension to specify the response methods for the event message that is generated when you commit a node in a workspace. The code for the node contains the MAX_PT function.
Deploy the code for the client that subscribes to event messages and the code for the extension, and run the code for the extension on your on-premises machine.
Download the demo project file:
- Environment requirement: Java 8 or later and Maven. Maven is a build automation tool for Java.
- Download link for the project file: extensions-demo-maxpt.zip (219 KB).
After you download the project file, enter the root directory of the project and run the following command:
```
mvn clean package -Dmaven.test.skip=true spring-boot:repackage
```
After you obtain the JAR package that can be directly installed, run the following command:
```
java -jar target/extensions-demo-maxpt-1.0.jar
```
If information in the following figure appears, the project is successfully started.Enter http://localhost:8080/index in the address bar of a browser and press Enter. If "hello world!" is returned, the extension is successfully deployed. You can subscribe to event messages after network connections are established between DataWorks and your extension and between EventBridge and your extension.

Verify the result

After you complete the preceding configurations, you can verify the result in the workspace. In this example, the extension can be triggered to check node commit events and node deployment events. You can configure the Parameters for Extension parameter to specify the workspace for which node commit events do not take effect. After you configure this parameter, the extension is not triggered to check the node commit event when you commit a node whose code contains the MAX_PT function in the specified workspace. However, the extension is triggered to check the node deployment event when you deploy a node whose code contains the MAX_PT function in the specified workspace.