An OSS object inspection node polls an OSS object based on FTP at a configured interval and triggers downstream nodes only after the object is detected. Use this node to coordinate DataWorks tasks with external scheduling systems when a downstream task must wait for an external write job to complete before it can proceed.
How it works
After an external scheduling system finishes writing to a database, it generates a marker file (for example,
XXXX2024-09-29.done) in the specified file system.An OSS data source reads the marker file from the file system.
The OSS object inspection node polls the OSS data source at the configured interval to check whether the marker file exists.
File detected: the node confirms the write job is complete and passes the result to its downstream node.
File not detected: the node fails the check and does not pass the result to its downstream node. It continues retrying at the configured interval until the condition to stop the detection is met.
Based on the detection result, the downstream node decides whether to access the external database.
File detected: the downstream node proceeds to access the database.
File not detected: the downstream node does not access the database.
The downstream node accesses the external database (Oracle, MySQL, SQL Server, and others).
The file system used to store marker files and the marker file format (for example, the .done suffix) are configurable.Prerequisites
Before you begin, ensure that you have:
A RAM user added to your workspace with the Develop or Workspace Administrator role assigned. The Workspace Administrator role grants broader permissions than typically needed — assign it with care. For details, see Add members to a workspace.
A serverless resource group associated with your workspace. For details, see Use serverless resource groups.
An OSS object inspection node created in Data Studio. For details, see Create a task node.
Permissions
The node uses Resource Access Management (RAM) and Security Token Service (STS) to authorize MaxCompute to access OSS in the development or production environment. The node runs under the permissions of a workspace administrator.
Before running the node, confirm the following:
MaxCompute has the required permissions on the OSS bucket. For details, see STS mode authorization.
The workspace administrator has the required permissions on the OSS bucket. For details, see Responsibilities of a workspace administrator.
Step 1: Configure the node
On the configuration tab of the OSS object inspection node, configure the following parameters:
| Parameter | Description |
|---|---|
| OSS Object | The OSS object path to monitor. Supports scheduling parameters such as ${bizdate} and ${cyctime}. |
| Timeout | The maximum time the node waits for the object to appear. When the timeout is reached, the node stops retrying. |
After configuring the detection parameters, set the scheduling properties for the node. For details, see Scheduling configurations.
Step 2: Deploy and monitor the node
Deploy the node to the production environment. For details, see Deploy nodes.
In Operation Center, go to the Auto Triggered Nodes page to view the deployed node and perform operations and maintenance (O&M). The system runs the node automatically based on the configured scheduling properties. For details, see Getting started with Operation Center.