DataWorks:Develop and deploy extensions based on Function Compute

Supported extension points

Extensions deployed on Function Compute currently support only the pre-event for data download.

When you register an extension, the Event and Applicable Service fields are populated automatically based on the extension point you select.

Prerequisites

Before you begin, make sure you have:

• A DataWorks Enterprise Edition subscription

• Access to the Function Compute console

• Read and write access to the Developer Backend. The following roles have this access: For more information, see Global module permission control and Manage product-level and console access with RAM policies.

  • Open Platform administrator

  • Tenant administrator

  • Alibaba Cloud accounts

  • RAM users with the AliyunDataWorksFullAccess policy attached

How it works

1. A user triggers an action in DataWorks that corresponds to a registered extension point event.

2. DataWorks sends the event message to your Function Compute function as an ExtensionRequest object.

3. Your function's handleRequest method processes the message and returns an ExtensionResponse with a CheckResult value.

4. DataWorks reads the CheckResult to decide whether to allow or block the operation.

After an extension point event is triggered, the associated process enters the Checking state and waits for the callback result. If your DataWorks Enterprise Edition subscription expires, all extensions become invalid and can no longer trigger event checks. In-progress checks that have not reached a final state automatically pass.

Limitations

• Only DataWorks Enterprise Edition users can use the Extensions module. If your subscription expires, all extensions become invalid and can no longer trigger event checks.

• Supported regions: China (Beijing), China (Hangzhou), China (Shanghai), China (Zhangjiakou), China (Shenzhen), China (Chengdu), US (Silicon Valley), US (Virginia), Germany (Frankfurt), Japan (Tokyo), China (Hong Kong), and Singapore.

• Multiple extensions can be associated with the same extension point event — a single event can trigger multiple extensions.

• When a check is triggered for a composite node that contains inner nodes — such as a machine learning (PAI) node, a do-while node, or a for-each node — all inner nodes must pass the check before subsequent operations can proceed.

Step 1: Configure extension dependencies

Add the following dependencies to your pom.xml file.

DataWorks dependency

<dependency>
  <groupId>com.aliyun</groupId>
  <artifactId>dataworks_public20200518</artifactId>
  <version>5.6.0</version>
</dependency>

Function Compute dependencies

<!-- https://mvnrepository.com/artifact/com.aliyun.fc.runtime/fc-java-core -->
<dependency>
    <groupId>com.aliyun.fc.runtime</groupId>
    <artifactId>fc-java-core</artifactId>
    <version>1.4.1</version>
</dependency>

<!-- https://mvnrepository.com/artifact/com.aliyun.fc.runtime/fc-java-event -->
<dependency>
    <groupId>com.aliyun.fc.runtime</groupId>
    <artifactId>fc-java-event</artifactId>
    <version>1.2.0</version>
</dependency>

Packaging configuration

Use the Apache Maven Shade plugin or the Apache Maven Assembly plugin to package all dependencies into a single JAR. The example below uses the Maven Shade plugin:

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>3.2.1</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <filters>
                            <filter>
                                <artifact>*:*</artifact>
                                <excludes>
                                    <exclude>META-INF/*.SF</exclude>
                                    <exclude>META-INF/*.DSA</exclude>
                                    <exclude>META-INF/*.RSA</exclude>
                                </excludes>
                            </filter>
                        </filters>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

Step 2: Write the extension code

An extension implements the PojoRequestHandler interface from the fc-java-core library. Your handleRequest method receives an ExtensionRequest object (the event message from DataWorks) and must return an ExtensionResponse with a CheckResult value.

A sample package is available for download and testing: fc_dataworks_demo01-1.0-SNAPSHOT.jar.

/project-root
    └── src
        └── main
            └── java
                └── com/aliyun/example
                    ├── App.java               (main handler — implements PojoRequestHandler)
                    ├── ExtensionRequest.java   (request model)
                    └── ExtensionResponse.java  (response model)
    └── pom.xml

package com.aliyun.example;

import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.JSONObject;
import com.aliyun.dataworks.ExtensionRequest;
import com.aliyun.dataworks.ExtensionResponse;
import com.aliyun.fc.runtime.Context;
import com.aliyun.fc.runtime.PojoRequestHandler;


public class App implements PojoRequestHandler<ExtensionRequest, ExtensionResponse> {

    public ExtensionResponse handleRequest(ExtensionRequest extensionRequest, Context context) {
        // Print the request content for debugging.
        System.out.println(JSON.toJSONString(extensionRequest));
        // Create a response object.
        ExtensionResponse extensionResponse = new ExtensionResponse();
        // Check whether eventType is 'upload-data-to-table'.
        if ("upload-data-to-table".equals(extensionRequest.getEventType())) {
            try {
                // Convert messageBody to a string and then parse it into a JSONObject.
                String messageBodyStr = JSON.toJSONString(extensionRequest.getMessageBody());
                JSONObject messageBody = JSON.parseObject(messageBodyStr);
                String tableGuid = messageBody.getString("tableGuid");
                // Check whether tableGuid contains 'ads'.
                if (tableGuid != null && tableGuid.contains("ads")) {
                    extensionResponse.setCheckResult("FAIL");
                } else {
                    extensionResponse.setCheckResult("OK");
                }
            } catch (Exception e) {
                extensionResponse.setCheckResult("FAIL");
                extensionResponse.setErrorMessage("Error processing request: " + e.getMessage());
                return extensionResponse;
            }
        } else {
            extensionResponse.setCheckResult("FAIL");
        }

        // Set an error message as feedback.
        extensionResponse.setErrorMessage("This is a test!");

        // Return the response object.
        return extensionResponse;
    }
}

public class ExtensionRequest {
    private Object messageBody;
    private String messageId;
    private String extensionBizId;
    private String extensionBizName;
    private String eventType;
    private String eventCategoryType;
    private Boolean blockBusiness;

    public ExtensionRequest() {
    }

    public Object getMessageBody() {
        return this.messageBody;
    }

    public void setMessageBody(Object messageBody) {
        this.messageBody = messageBody;
    }

    public String getMessageId() {
        return this.messageId;
    }

    public void setMessageId(String messageId) {
        this.messageId = messageId;
    }

    public String getExtensionBizId() {
        return this.extensionBizId;
    }

    public void setExtensionBizId(String extensionBizId) {
        this.extensionBizId = extensionBizId;
    }

    public String getExtensionBizName() {
        return this.extensionBizName;
    }

    public void setExtensionBizName(String extensionBizName) {
        this.extensionBizName = extensionBizName;
    }

    public String getEventType() {
        return this.eventType;
    }

    public void setEventType(String eventType) {
        this.eventType = eventType;
    }

    public String getEventCategoryType() {
        return this.eventCategoryType;
    }

    public void setEventCategoryType(String eventCategoryType) {
        this.eventCategoryType = eventCategoryType;
    }

    public Boolean getBlockBusiness() {
        return this.blockBusiness;
    }

    public void setBlockBusiness(Boolean blockBusiness) {
        this.blockBusiness = blockBusiness;
    }
}

public class ExtensionResponse {
    private String checkResult;
    private String errorMessage;

    public ExtensionResponse() {
    }

    public String getCheckResult() {
        return this.checkResult;
    }

    public void setCheckResult(String checkResult) {
        this.checkResult = checkResult;
    }

    public String getErrorMessage() {
        return this.errorMessage;
    }

    public void setErrorMessage(String errorMessage) {
        this.errorMessage = errorMessage;
    }
}

mvn clean package

Step 3: Deploy to Function Compute

1. Log on to the Function Compute console. In the left navigation pane, click Tasks.

2. Click Create Function to create a Function Compute service and function. For more information, see Create a service and Create a function.

   • Set the runtime based on your code. If you use the sample package fc_dataworks_demo01-1.0-SNAPSHOT.jar, set Runtime to Java 8. Download the package to your local machine and upload it to Code Package.

   • For more information about function management, see Manage functions.

   

3. Configure the handler (function entry point). For Java functions, use the format [Package name].[Class name]::[Method name]. For example:

   The default handler for Function Compute is example.App::handleRequest. Update this to match your own code. For more information, see Handler.

   Component	Example value
   Package name	example
   Class name	HelloFC
   Method name	handleRequest
   Handler string	example.HelloFC::handleRequest

4. Test the function. On the function details page, select the Function Code tab and click Test Function. After the Execution Successful message appears, proceed to register the extension in DataWorks.

Step 4: Register the extension in DataWorks

1. Log on to the DataWorks console. In the top navigation bar, select the region. In the left navigation pane, choose More > Open Platform, then click Go to Open Platform. The Developer Backend tab appears.

2. In the left navigation pane, click Extensions.

3. Click Extension List > Register Extension, select Deploy With Function Compute, and configure the parameters.

   Parameter	Description
   Extension name	A custom name for the extension
   Function Compute service and function	The Function Compute service and function where the extension is deployed. DataWorks sends extension event messages directly to this service
   Processed extension points	The extension point events to handle. Currently supported: pre-events for data download, asset listing and delisting, and data upload. The Event and Applicable Service fields are automatically populated after selection
   Owner	The owner of the extension, so users know who to contact if they encounter issues
   Workspace for testing	A workspace for end-to-end testing before publishing. Developers can trigger events to verify that DataWorks sends messages via EventBridge and that the extension receives, reviews, and responds with a callback. Not required for tenant-level extension point events
   Extension details address	URL of a page with details about the extension — for example, the check path and the reason for blocking when an extension check is triggered
   Extension documentation address	URL of the help documentation for the extension, so users can learn about the check logic and properties
   Extension parameter configuration	Parameters used in your extension code, in key=value format (one per line). Supports built-in DataWorks parameters and custom parameters. For details, see Advanced feature: Configure extension parameters
   Extension options configuration	A JSON string defining configurable items that allow workspace-level customization of the extension. For details, see Advanced feature: Configure extension options

4. Click OK to complete registration. The extension appears in the Extension List.

Step 5: Publish the extension

After the extension passes testing and approval, administrators other than the extension owner can enable it in the Management Center. For more information, see Apply extensions.

Appendix: Format of messages sent from DataWorks to Function Compute

All event messages share the following structure. The messageBody field contains the event-specific content, which varies by message type.

{
  "blockBusiness": true,
  "eventCategoryType": "resources-download",
  "eventType": "upload-data-to-table",
  "extensionBizId": "job_6603643923728775070",
  "messageBody": {
    "tenantId": 28378****10656,
    "eventCode": "xxxx"
  },
  "messageId": "52d44ee7-b51f-4d4d-afeb-*******"
}

For the full event message content by message type, see Development reference: Event list and message formats.

What's next

• Development reference: Event list and message formats — full list of event types and their messageBody schemas

• Apply extensions — enable an extension across workspaces after publishing

• Develop and deploy an extension based on a self-managed service — alternative to Function Compute if you need a self-hosted runtime

• What is Function Compute? — overview of the Function Compute service