An FTP Check node can be used to periodically detect whether a specific file exists based on File Transfer Protocol (FTP). If the FTP Check node detects that the file exists, the scheduling system starts to run the descendant node of the FTP Check node. Otherwise, the FTP Check node retries the detection based on the configured detection interval. The FTP Check node stops the retry until the condition for stopping the detection is met. In most cases, FTP Check nodes are used for communications between the DataWorks scheduling system and external scheduling systems. This topic describes how to use an FTP Check node and the related precautions.
Prerequisites
- An FTP data source is added. For more information, see Add an FTP data source.
- An exclusive resource group for scheduling is created. For more information, see Performance metrics and billing standards of exclusive resource groups for scheduling.
- A workflow is created. For more information, see Create a workflow.
Background information
.done
in the file system to indicate that the data write task is completed. Then, you can
create an FTP Check node in the DataWorks scheduling system to periodically detect
whether the marker file with the suffix .done
exists. If the file exists, the node that needs to access the external database can
be scheduled.
- You can specify the file system that can be used to store the marker files.
- In this example, a marker file with the suffix
.done
is used. You can customize the information such as the format and name for your marker file.
Limits
- Only the China (Beijing), China (Shanghai), China (Hangzhou), China (Shenzhen), China (Zhangjiakou), China (Chengdu), and Singapore (Singapore) regions support FTP Check nodes.
- FTP Check nodes can run only on exclusive resource groups for scheduling.
- If an FTP Check node is scheduled by minute or hour, you can set the Check stop policy parameter only to Number of Check stops for the node.