Integrate External Workflows via Cross-Platform Trigger Nodes - Dataphin

When you build a data system, you may have nodes with dependencies across different platforms. You can use trigger nodes to manage these dependencies. For example, an algorithm is developed in System A, and a recurring task in Dataphin must run based on the output from that algorithm. This topic describes how to develop trigger nodes.

Prerequisites

The following prerequisites must be met to use external trigger nodes:

You have purchased the Dataphin OpenAPI feature.
The environment that sends the trigger request must have network connectivity to Dataphin.

Basic characteristics of trigger nodes

Trigger nodes and other compute nodes have the following similarities and differences:

A recurring instance of a trigger node starts to run when the following conditions are met:
1. The current system time is later than or equal to the scheduled runtime of the instance. This is the same as other compute nodes.
2. As with other compute tasks, the runtime status of all ancestor nodes is Succeeded.
3. An external trigger request has been received. This condition is unique to trigger nodes.
Data backfill instances for trigger nodes do not depend on external trigger requests. This is the same as for other compute nodes.

Trigger Methods

Dataphin supports sending trigger requests from an external system only through the OpenAPI. This is called an external trigger.

External trigger

Follow these steps to use an external trigger:

Obtain the trigger information.
Before you send an external trigger request, you must obtain the following information.
- Node ID (node_id): Find the node ID in the Basic Information section of the Properties panel for the submitted trigger node.
- Project ID (project_id): Find the project ID in the URL of your browser on the Develop page. For example, if the URL is https://www.aliyun.com/dev/codeManage/698348****632?env=DEV&projectId=6982117****30048&tenantId=309107012, the project ID is 6982117****30048.
- Tenant ID (tenant_id): Find the tenant ID in the URL of your browser on the Develop page. For example, if the URL is https://www.aliyun.com/dev/codeManage/698348****632?env=DEV&projectId=6982117****30048&tenantId=309***012, the tenant ID is 309***012.

Write the trigger node code.

This topic uses Java code as an example.

Important

Obtain the required JAR packages from the Alibaba Cloud helpdesk.

DefaultProfile profile = DefaultProfile.getProfile("<Your region>",
                "<Dataphin accessKey>", "<Dataphin secret>");
        DefaultProfile.addEndpoint("<Your region>", "dataphin-public", "<Your openapi endpoint>");
        profile.getHttpClientConfig().setIgnoreSSLCerts(true);
        
        DefaultAcsClient client =  new DefaultAcsClient(profile);
        ExecuteTriggerNodeRequest request = new ExecuteTriggerNodeRequest();
        // Set the ID of the tenant where the node to be triggered is located.
        request.setOpTenantId(0L);
        // Set the data timestamp for the recurring instance to be triggered. The format is yyyy-MM-dd.
        request.setBizDate("<bizdate>");
        // Set the ID of the node to be triggered.
        request.setNodeId("<node_id>");
        // Set the ID of the project where the node to be triggered is located.
        request.setProjectId(0L);
        // Set the environment type.
        request.setEnv("PROD");
        ExecuteTriggerNodeResponse response = client.getAcsResponse(request);
        System.out.println(response.getCode());

After the trigger node code runs successfully and Dataphin receives the trigger request, the status of the trigger node changes to Succeeded.