This topic describes the parameters that FTP Writer supports and how to configure it by using the codeless user interface (UI) and code editor.

FTP Writer allows you to write one or more comma-separated values (CSV) files to a remote FTP server. FTP Writer converts the data that is obtained from a Data Integration reader to CSV files. Then, FTP Writer writes these files to a remote FTP server by using FTP-related network protocols.
Note You must configure an FTP connection before you configure FTP Writer. For more information, see Configure an FTP connection.

FTP Writer can write files that store logical two-dimensional tables, such as CSV files that store text data, to an FTP server.

FTP Writer allows you to convert the data that is obtained from a Data Integration reader to files and write these files to an FTP server. The files on the FTP server store only unstructured data. FTP Writer supports the following features:
  • Writes only files that store text data. The text data must be logical two-dimensional tables. FTP Writer cannot write files that store Binary Large Object (BLOB) data such as video data.
  • Writes CSV-like and text files with custom delimiters.
  • Writes compressed files to an FTP server.
  • Uses concurrent threads to write files. Each thread writes a file.
FTP Writer does not support the following features:
  • Uses concurrent threads to write a single file.
  • Distinguishes between data types. FTP does not distinguish between data types. Therefore, FTP Writer writes all data as strings to files on an FTP server.

Parameters

Parameter Description Required Default value
datasource The connection name. It must be the same as the name of the added connection. You can add connections in the code editor. Yes N/A
timeout The timeout period to connect to the FTP server. Unit: milliseconds. No 60,000
path The directory on the FTP server to which files are written. FTP Writer concurrently writes multiple files to the directory based on the concurrency setting. Yes N/A
fileName The name prefix of the files to be written to the FTP server. A random suffix is appended to the specified prefix to form the actual file name that is used by each thread. Yes N/A
writeMode The mode in which FTP Writer writes files. Valid values:
  • truncate: FTP Writer deletes all existing files with the specified file name prefix in the destination directory before it writes files to the directory.
  • append: FTP Writer directly writes files based on the specified file name prefix and ensures that the actual file names do not conflict with those of existing files.
  • nonConflict: FTP Writer returns an error if a file with the specified file name prefix exists in the destination directory.
Yes N/A
fieldDelimiter The column delimiter that is used in the files to be written to the FTP server. The delimiter must be a single character. Yes N/A
skipHeader Specifies whether to skip the header (if exists) of a CSV-like file. The skipHeader parameter is not supported for compressed files. No false
compress The compression format of the files to be written to the FTP server. The GZIP and BZIP2 compression formats are supported. No None
encoding The encoding format of the files to be written to the FTP server. No utf-8
nullFormat The string that represents null. No standard strings can represent null in text files. Therefore, Data Integration provides the nullFormat parameter to define a string that represents a null pointer.

For example, if you specify nullFormat="null", Data Integration considers null as a null pointer.

No N/A
dateFormat The format in which the data of the DATE type is serialized in a file, for example, "dateFormat":"yyyy-MM-dd". No N/A
fileFormat The format in which files are written to the FTP server. Valid values: csv and text. If a file is written as a CSV file, the file follows CSV specifications. If the data in the file contains the column delimiter, the column delimiter is escaped by using double quotation marks (" "). If a file is written as a text file, the data in the file is separated with the column delimiter. If the data in the file contains the column delimiter, the column delimiter is not escaped. No TEXT
header The table header if files are written as text files, for example, ['id', 'name', 'age']. No N/A
markDoneFileName The name of the file whose existence indicates that the sync node is properly run. Data Integration checks whether the file exists after data synchronization. Set this parameter to the absolute path of the file. No N/A

Configure FTP Writer by using the codeless UI

  1. Configure the connections.
    Configure the connections to the source and destination data stores for the sync node. Select data source section
    Parameter Description
    Data source The datasource parameter in the preceding parameter description. Select a connection type and select the name of a connection that you have configured in DataWorks.
    File path The path parameter in the preceding parameter description.
    Text Type The format of the files to be written to the FTP server. Default value: csv.
    Column separator The fieldDelimiter parameter in the preceding parameter description. The default delimiter is comma (,).
    Encoding format The encoding parameter in the preceding parameter description. Default value: UTF-8.
    null value The nullFormat parameter in the preceding parameter description, which defines a string that represents null.
    Time format The dateFormat parameter in the preceding parameter description.
    Prefix conflict The writeMode parameter in the preceding parameter description.
  2. Configure field mapping. Fields in the source table on the left have a one-to-one mapping with fields in the destination table on the right.Field Mapping section
    GUI element Description
    The same name mapping Click The same name mapping to establish a mapping between fields with the same name. Note that the data types of the fields must match.
    Peer mapping Click Peer mapping to establish a mapping between fields in the same row. Note that the data types of the fields must match.
    Unmap Click Unmap to remove mappings that have been established.
  3. Configure channel control policies.Channel control section
    Parameter Description
    Maximum number of concurrent tasks expected The maximum number of concurrent threads that the sync node uses to read data from or write data to data stores. You can configure the concurrency for the node on the codeless UI.
    Synchronization rate Specifies whether to enable bandwidth throttling. You can enable bandwidth throttling and set a maximum transmission rate to avoid heavy read workload of the source. We recommend that you enable bandwidth throttling and set the maximum transmission rate to a proper value.
    The number of error records exceeds The maximum number of dirty data records allowed.

Configure FTP Writer by using the code editor

You can configure FTP Writer when you use the code editor to configure a sync node to write files to an FTP server. For more information, see Create a sync node by using the code editor.
{
    "type":"job",
    "version":"2.0", // The version number.
    "steps":[
        { 
            "stepType":"stream",
            "parameter":{},
            "name":"Reader",
            "category":"reader"
        },
        {
            "stepType":"ftp",// The writer type.
            "parameter":{
                "path":"",// The directory on the FTP server to which files are written.
                "fileName":"",// The name prefix of the files to be written to the FTP server.
                "nullFormat":"null",// The string that represents null.
                "dateFormat":"yyyy-MM-dd HH:mm:ss",// The format of the time.
                "datasource":"",// The connection name.
                "writeMode":"",// The write mode.
                "fieldDelimiter":",",// The column delimiter.
                "encoding":"",// The encoding format.
                "fileFormat":""// The format in which FTP Writer writes files.
            },
            "name":"Writer",
            "category":"writer"
        }
    ],
    "setting":{
        "errorLimit":{
            "record":"0"// The maximum number of dirty data records allowed.
        },
        "speed":{
            "throttle":false,// Specifies whether to enable bandwidth throttling. A value of false indicates that the bandwidth is not throttled. A value of true indicates that the bandwidth is throttled. The maximum transmission rate takes effect only if you set this parameter to true.
            "concurrent":1 // The maximum number of concurrent threads.
        }
    },
    "order":{
        "hops":[
            {
                "from":"Reader",
                "to":"Writer"
            }
        ]
    }
}