DataWorks:FTP Check node

How it works

The typical pattern is for an external system to write data to a database, then create a marker file (for example, xxxx2021-03-03.done) on an FTP server to signal that the data is ready.

The process works as follows:

1. The external system generates a marker file when data is ready. The external scheduling system determines the file system for the marker file.

2. The FTP data source reads the marker file from the file system.

3. The FTP Check node polls the data source for the marker file based on the configured check strategy:

   • Found: The node returns success and triggers the downstream node.

   • Not found: The node retries according to the configured strategy until the limit is reached.

4. The downstream node runs only if the FTP Check node returns success:

   • Success: The downstream node accesses the external database (for example, Oracle, MySQL, or SQL Server).

   • Failure: The downstream node does not run.

The .done file in this example is illustrative — use any filename that fits your workflow.

Prerequisites

Before you begin, make sure you have:

• An FTP data source. SFTP configurations that use Key authentication are not supported.

• A workflow. For details, see Create an auto triggered workflow.

Limitations

Create an FTP Check node

1. Log on to the DataWorks console.

2. Go to DataStudio. Hover over the icon and select General > FTP Check. Alternatively, open the desired workflow, right-click General, and select Create > FTP Check.

3. In the Create Node dialog box, enter the Name and select the Target Folder, then click Confirm.

   The node name can contain letters, Chinese characters, digits, underscores (_), and periods (.). Maximum length: 128 characters.

4. In the right pane, click Properties to configure scheduling. Configure Basic Properties, Time Properties, Resource Properties, and Dependencies. For details, see Configure basic properties, Configure time properties, Configure the resource property, and Configure same-cycle scheduling dependencies.

5. Configure the detection object and check strategy.

   1. From the Select FTP Data Source list, select the target data source. You can select an FTP or SFTP data source. If no data source is available, create one. For details, see Add an FTP data source.

   2. In Check File, enter the file path to poll. If the path is dynamic, you can use scheduling parameters to configure a variable path. For all supported scheduling parameter formats, see Supported formats for scheduling parameters.

   3. In Check Interval (s), set the polling interval in seconds.

   4. In Check Stop Strategy, choose when the node stops polling:

      Strategy	Supported scheduling cycles	How it works
      Check Stop Time	Day only (not supported for Minute or Hour)	Stops at a fixed cutoff time (hh24:mi:ss). If the file is not found by that time, the task fails. If you switch from a Day cycle to Minute or Hour, change this to Check Stop Count before committing.
      Check Stop Count	All cycles, including Minute and Hour	Stops after a fixed number of polling attempts. If the file is not found after N attempts, the task fails.

   

6. Save and commit the node.

   1. Click the icon to save the node.

   2. Click the icon. In the Submit New Version dialog box, enter a Change Description and click Confirm.

   3. For standard mode workspaces, click Publish after committing. For details, see Publish tasks.

   Configure the Rerun and Upstream Dependencies properties before committing.

7. Perform O&M operations on the node. For details, see Perform basic O&M operations on auto triggered nodes.

What's next

• To poll for a file using a more modern approach, use Check nodes.

• To configure how downstream nodes depend on this node, see Configure same-cycle scheduling dependencies.