This topic describes the parameters that are supported by FTP Writer and how to configure FTP Writer by using the codeless user interface (UI) and code editor.

FTP Writer writes one or more comma-separated values (CSV) files to a remote File Transfer Protocol (FTP) server. FTP Writer converts the data that is obtained from a reader to CSV files. Then, FTP Writer writes these files to a remote FTP server by using FTP-related network protocols.
Note Before you configure FTP Writer, you must add an FTP data source. For more information, see Add an FTP data source.

FTP Writer can write files that store logical two-dimensional tables, such as CSV files that store text data, to an FTP server.

FTP Writer converts the data that is obtained from a reader to files and writes the files to an FTP server. The files on the FTP server store only unstructured data. FTP Writer provides the following features:
  • Writes only text files to an FTP server. The data in the files must be organized as logical two-dimensional tables. FTP Writer cannot write files that store binary large object (BLOB) data, such as video data, to an FTP server.
  • Writes TXT and CSV-like files that contain custom delimiters to an FTP server.
  • Writes uncompressed files to an FTP server.
  • Uses parallel threads to write files to an FTP server. Each thread writes a file.
FTP Writer does not support the following features:
  • Uses parallel threads to write a single file to an FTP server.
  • Distinguishes between data types. FTP does not distinguish between data types. Therefore, FTP Writer writes all data as strings to files on an FTP server.

Parameters

Parameter Description Required Default value
datasource The name of the data source. It must be the same as the name of the added data source. You can add data sources by using the code editor. Yes No default value
timeout The timeout period of the connection to an FTP server. Unit: milliseconds. No 60,000
path The directory on the FTP server to which you want to write data. FTP Writer uses parallel threads to write multiple files to the directory based on the parallelism setting. Yes No default value
fileName The name prefix of the files that you want to write to the FTP server. A random suffix is appended to the specified prefix to form the actual file name that is used by each thread. Yes No default value
singleFileOutput Specifies whether to add a random suffix to the names of the files that you want to write to the FTP server. The names of the files that FTP Writer writes to the FTP server are related to the value of the fileName parameter. If you do not need the random suffix, set the singleFileOutput parameter to true. In this case, FTP Writer the files to the FTP server by using the name prefix of the files. No false
writeMode The mode in which FTP Writer writes files. Valid values:
  • truncate: FTP Writer deletes all existing files whose names contain the prefix specified by fileName in the destination directory before it writes files to the directory.
  • append: FTP Writer directly writes files based on the file name prefix specified by fileName and ensures that the actual file names do not conflict with the names of existing files.
  • nonConflict: FTP Writer returns an error if the destination directory contains a file whose name contains the prefix specified by fileName.
Yes No default value
fieldDelimiter The column delimiter that is used in the files that you want to write to the FTP server. The delimiter must be a single character. Yes No default value
skipHeader Specifies whether to skip the headers in CSV-like files if the files contain headers. The skipHeader parameter is not supported for compressed files. No false
compress The compression format of the files that you want to write to the FTP server. Valid values: gzip and bzip2. No Not default value
encoding The encoding format of the files that you want to write to the FTP server. No utf-8
nullFormat The string that represents a null pointer. No standard strings can represent a null pointer in TXT files. You can use this parameter to define which string represents a null pointer.

For example, if you set nullFormat to null, Data Integration considers null as a null pointer.

No No default value
dateFormat The format in which the data of the DATE type is serialized in a file, such as "dateFormat":"yyyy-MM-dd". No No default value
fileFormat The format in which files are written to the FTP server. Valid values: CSV and TEXT. If a file is written as a CSV file, the file must follow CSV specifications. If the data in the file contains column delimiters, the column delimiters are escaped by double quotation marks ("). If a file is written as a TXT file, the data in the file is separated by column delimiters. In this case, the column delimiters are not escaped. No TEXT
header The table headers if files are written as TXT or CSV files, such as ["id","name","age"]. This indicates that the id, name, and age fields are written to a CSV file as the first row. No No default value
markDoneFileName The name of the file that is used to indicate that the synchronization node is successfully run. Data Integration checks whether the file exists after data synchronization. Set this parameter to the absolute path of the file. No No default value

Configure FTP Writer by using the codeless UI

  1. Configure data sources.
    Configure Source and Target for the synchronization node. Configure data sources
    Parameter Description
    Connection The name of the data source to which you want to write data. This parameter is equivalent to the datasource parameter that is described in the preceding section.
    File Path This parameter is equivalent to the path parameter that is described in the preceding section.
    File Type The format of the files that you want to write to the FTP server. The default format is csv.
    Field Delimiter This parameter is equivalent to the fieldDelimiter parameter that is described in the preceding section. By default, a comma (,) is used as a column delimiter.
    Encoding This parameter is equivalent to the encoding parameter that is described in the preceding section. Default value: UTF-8.
    Null String This parameter is equivalent to the nullFormat parameter that is described in the preceding section, which defines a string that represents a null pointer.
    Time Format This parameter is equivalent to the dateFormat parameter that is described in the preceding section.
    Solution to Duplicate Prefixes This parameter is equivalent to the writeMode parameter that is described in the preceding section.
  2. Configure field mappings. This operation is equivalent to setting the column parameter that is described in the preceding section. Fields in the source on the left have a one-to-one mapping with fields in the destination on the right. Field mappings
    Operation Description
    Map Fields with the Same Name Click Map Fields with the Same Name to establish mappings between fields with the same name. The data types of the fields must match.
    Map Fields in the Same Line Click Map Fields in the Same Line to establish mappings between fields in the same row. The data types of the fields must match.
    Delete All Mappings Click Delete All Mappings to remove the mappings that are established.
  3. Configure channel control policies. Channel control
    Parameter Description
    Expected Maximum Concurrency The maximum number of parallel threads that the synchronization node uses to read data from the source or write data to the destination. You can configure the parallelism for the synchronization node on the codeless UI.
    Bandwidth Throttling Specifies whether to enable bandwidth throttling. You can enable bandwidth throttling and specify a maximum transmission rate to prevent heavy read workloads on the source. We recommend that you enable bandwidth throttling and set the maximum transmission rate to an appropriate value based on the configurations of the source.
    Dirty Data Records Allowed The maximum number of dirty data records that are allowed.

Configure FTP Writer by using the code editor

In the following code, a synchronization node is configured to write files to an FTP server. For more information about how to configure a synchronization node by using the code editor, see Create a sync node by using the code editor.
{
    "type":"job",
    "version":"2.0",// The version number. 
    "steps":[
        { 
            "stepType":"stream",
            "parameter":{},
            "name":"Reader",
            "category":"reader"
        },
        {
            "stepType":"ftp",// The writer type. 
            "parameter":{
                "path":"",// The directory on the FTP server to which you want to write files. 
                "fileName":"",// The name prefix of the files that you want to write to the FTP server. 
                "nullFormat":"null",// The string that represents a null pointer. 
                "dateFormat":"yyyy-MM-dd HH:mm:ss",// The time format. 
                "datasource":"",// The name of the data source. 
                "writeMode":"",// The write mode. 
                "fieldDelimiter":",",// The column delimiter. 
                "encoding":"",// The encoding format. 
                "fileFormat":""// The format in which FTP Writer writes files. 
            },
            "name":"Writer",
            "category":"writer"
        }
    ],
    "setting":{
        "errorLimit":{
            "record":"0"// The maximum number of dirty data records allowed. 
        },
        "speed":{
            "throttle":true,// Specifies whether to enable bandwidth throttling. The value false indicates that bandwidth throttling is disabled, and the value true indicates that bandwidth throttling is enabled. The mbps parameter takes effect only when the throttle parameter is set to true. 
            "concurrent":1, // The maximum number of parallel threads. 
            "mbps":"12"// The maximum transmission rate.
        }
    },
    "order":{
        "hops":[
            {
                "from":"Reader",
                "to":"Writer"
            }
        ]
    }
}