DataWorks:Develop and deploy extensions based on self-managed services

Prerequisites

You must enable message subscription. The deployment of extensions using a self-hosted service relies on the message distribution capabilities of EventBridge. You must ensure that DataWorks open event messages are configured to be sent to an event bus in EventBridge. You must also ensure that messages from this event bus are routed to a service program that is deployed on-premises or in the cloud.

Limits

• Only users of DataWorks Enterprise Edition can use the Extensions module.

• The Extensions module is available in the following regions: China (Beijing), China (Hangzhou), China (Shanghai), China (Zhangjiakou), China (Shenzhen), China (Chengdu), US (Silicon Valley), US (Virginia), Germany (Frankfurt), Japan (Tokyo), China (Hong Kong), and Singapore.

Notes

• Only the Open Platform administrator, tenant administrator, Alibaba Cloud accounts, and RAM users to which the AliyunDataWorksFullAccess policy is attached have read and write permissions on the developer backend. For more information about permission management, see Global module permission control and Manage product-level and console access with RAM policies.

• Version limit: If your DataWorks Enterprise Edition subscription expires, all extensions become invalid and can no longer trigger event checks. Any checks that have been triggered but have not reached their final state will automatically pass.

• Node limit: When a check is triggered for a composite node that contains inner nodes, such as a machine learning (PAI) node, a do-while node, or a for-each node, all inner nodes must pass the check before subsequent operations can proceed.

• Trigger description: Multiple extensions can be associated with the same extension point event. This means a single event can trigger multiple extensions.

Step 1: Configure extension dependencies

When you develop an extension, add the following dependencies to the pom.xml file. EventBridge supports various types of endpoints to process and consume events. In addition to the following dependencies, you can configure other dependencies based on the event target set in EventBridge and the final message routing.

<dependency>
 <groupId>com.aliyun</groupId>
 <artifactId>dataworks_public20200518</artifactId>
 <version>5.6.0</version>
</dependency>

<build>
        <plugins>
              <plugin>
                  <groupId>org.apache.maven.plugins</groupId>
                  <artifactId>maven-shade-plugin</artifactId>
                  <version>3.2.1</version>
                  <executions>
                    <execution>
                      <phase>package</phase>
                      <goals>
                        <goal>shade</goal>
                      </goals>
                      <configuration>
                        <filters>
                          <filter>
                            <artifact>*:*</artifact>
                            <excludes>
                              <exclude>META-INF/*.SF</exclude>
                              <exclude>META-INF/*.DSA</exclude>
                              <exclude>META-INF/*.RSA</exclude>
                            </excludes>
                          </filter>
                        </filters>
                      </configuration>
                    </execution>
                  </executions>
              </plugin>
        </plugins>
</build>

Step 2: Develop the extension code

Messages routed by the event bus in EventBridge are pushed to a service that is deployed on-premises or in the cloud. This service receives the DataWorks messages that are pushed by the event bus and uses a specific DataWorks API to send a callback with the processed result.

1. Develop the program code.

   Parse the message content

   For the format of event messages that are pushed by DataWorks, see Appendix: Format of messages sent from DataWorks to EventBridge. In the message format, data contains the specific message content. During development, you can use the data.eventCode field to identify the message type and the id field to retrieve message details.

   Note

   OpenEvent uses EventBridge to distribute DataWorks event messages. Before you develop an extension, you must subscribe to DataWorks messages in EventBridge. For more information, see Enable message subscription.

   Write the processing logic

   Process the messages that are pushed by the event bus as needed. During extension development, you can use the following methods to improve development efficiency and application performance.

   • Use Advanced feature: Configure extension parameters, such as extension.project.disabled, to disable the extension for a specific workspace.

   • When you process extension points that are related to the Data Studio module, you can call the GetIDEEventDetail API to retrieve a data snapshot from the time when the extension point event was triggered based on the MessageId.

   Note

   MessageId corresponds to the id field in the message. For more information, see Appendix: Format of messages sent from DataWorks to EventBridge.

   Return the processing result to DataWorks

   The extension service must return the processing result for the extension point to DataWorks through an OpenAPI. When you make the callback, you must select the appropriate OpenAPI based on the module where the extension point event occurred.

   • Extension point events in DataStudio: Use the UpdateIDEEventResult API to send the callback that contains the processing result.

   • Extension point events in Operation Center: Use the UpdateWorkbenchEventResult API to send the callback that contains the processing result.

   • Extension point events in other modules: Use the CallbackExtension API to send the callback that contains the processing result.

   The callback API returns the extension (ExtensionCode), event message (MessageId), and message processing result (CheckResult) for the current service.

   • CheckResult values:

     • OK: The extension passed the check for this extension point event.

     • FAIL: The extension failed the check for this extension point event. You must view and handle the error in a timely manner to prevent impacts on subsequent program execution.

     • WARN: The extension passed the check for this extension point event, but with a warning.

   • ExtensionCode: You can obtain this code from the extension list page in DataWorks after you register the extension as described in the following sections.

   • MessageId: Corresponds to the ID field in the message. For more information, see Format of messages sent from DataWorks to EventBridge.

2. After the code is developed, you can package the program into a runnable .jar file for subsequent service deployment.

Step 3: Deploy the extension

After you develop and debug the extension code, you can deploy the packaged code as an application service on Alibaba Cloud ECS or another service provider.

Step 1: Register the extension

Before you can develop and use an extension, you must register it in DataWorks to obtain the corresponding Extension Code for subsequent development. The following procedure describes how to register an extension.

1. Go to the Developer Backend tab.

   Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose More > Open Platform. On the page that appears, click Go to Open Platform. The Developer Backend tab appears.

2. Register the extension.

   1. In the left navigation pane, click Extensions to open the Extensions page.

   2. Click Extension List > Register Extension. Select Deploy with a self-hosted service and configure the extension details.

      Parameter description

      Parameter	How to configure
      Extension Name	A custom name for the extension, used for identification.
      Extension Points to Process	Select the types of extension event messages that this extension will process. For a list of supported extension points, see List of supported extension point events. Select extension points as needed. Note After you make a selection, the interface automatically matches the Event and Applicable Module. No manual configuration is required. Extension point event limits: Extension point events are divided into tenant-level and workspace-level. When you register an extension, you can only select one level of extension point events. For the level of each supported extension point event in DataWorks, see Development Reference: Event list and message format. Extensions deployed using Function Compute currently support only the Pre-Data-Download Event.
      Owner	The owner of the extension. This allows users to contact the owner if they encounter problems.
      Test Workspace	Select a workspace to test the extension. The extension takes effect in the test workspace without being published. Before the extension is published, developers can perform end-to-end testing and validation in the test workspace. By triggering events, they can test whether DataWorks sends messages through EventBridge and whether the extension receives, reviews, and sends callbacks for the messages. Note If you select tenant-level extension point events for Extension Points to Process, you do not need to configure a Test Workspace.
      Extension Details URL	Enter the URL of a page that provides details about the extension. This helps users better understand and use the extension. You can develop a details page for the extension and configure its URL here. This allows users to view the complete validation process, such as the check path and the reason for blocking, when the extension is triggered.
      Extension Document URL	Enter the URL of the help document for the extension. You can develop a help document page for the extension and configure its URL here. This allows users to learn about the extension's validation logic and properties.
      Extension Parameter Settings	DataWorks supports using parameters during development to improve efficiency. Add the parameters you need to use in the extension code here. You can use built-in parameters for common scenarios provided by DataWorks, or you can define your own. You can add multiple parameters, one per line, in the key=value format. Note For example, you can use the built-in parameter extension.project.disabled to disable the extension for a specific workspace. For more information about using parameters, see Advanced feature: Configure extension parameters.
      Extension Option Settings	Enter the configuration items for extension users. This allows for personalized control of the extension in different workspaces. The extension developer must define the options here as a JSON string. For example, you can use option settings to let users control the SQL length themselves. For the JSON format, see Advanced feature: Configure extension option settings.

3. Complete the extension registration.

   Click OK to finish.

   Note

   After a successful registration, you can view the extension in the Extension List.

Step 2: Publish the extension

After the extension is developed, deployed, and registered in DataWorks, you must complete the testing, approval, and publishing process. Then, administrators other than the extension owner can enable the extension in the Management Center. For more information, see Apply extensions.

{
    "datacontenttype": "application/json;charset=utf-8", // The format of the content in the data parameter. datacontenttype supports only the application/json format.
    "aliyunaccountid": "1111",// The ID of the Alibaba Cloud account.
    "aliyunpublishtime": "2024-07-10T07:25:34.915Z",// The time when EventBridge received the event.
    "data": {
              
             "tenantId": 28378****10656,// The tenant ID. Each Alibaba Cloud account corresponds to a tenant in DataWorks, and each tenant has its own ID. You can find this value in the user information section in the upper-right corner of DataWorks Data Studio.
             "eventCode": "xxxx"// The event code.
            
    },
    "aliyunoriginalaccountid": "11111",
    "specversion": "1.0",
    "aliyuneventbusname": "default",// The name of the EventBridge event bus used to receive DataWorks event messages.
    "id": "45ef4dewdwe1-7c35-447a-bd93-fab****",// The event ID. A unique value that identifies the event.
    "source": "acs.dataworks",// The event source, which is the service that provides the event. This indicates that the message was pushed by DataWorks.
    "time": "2024-07-10T15:25:34.897Z",// The time when the event occurred.
    "aliyunregionid": "cn-shanghai",// The region where the event was received.
    "type": "dataworks:ResourcesUpload:UploadDataToTable"// The event type. You can use this event type in the EventBridge console to filter all messages pushed by DataWorks. The Type value is different for each event.
}