All Products
Document Center

Configure FTP writer

Last Updated: Apr 03, 2018

FTP Writer is used to write one or more files to a remote FTP file in a csv format. At the underlying implementation level, FTP Writer converts the data under Data Integration transfer protocol to csv files and writes these files to the remote FTP server using FTP‑related network protocols. You must configure the data source before configuring the FTP Writer plug‑in. For more information, see FTP data source config.

What is written and saved to the FTP file is a two‑dimensional table in a logic sense, for example, text information in a csv format.

FTP Writer provides the ability to convert the Data Integration protocol to a FTP file, which itself is a non‑structured data storage. FTP Writer supports the following features:

  • Only supports writing text files (BLOB is not supported, for example, video data) and schema in the text file must be a two-dimensional table.

  • Supports csv and text files with custom delimiters.

  • Does not support text compression during writing.

  • Supports multi-thread writing, with different subfiles written using different threads.

Currently unsupported features:

  • Concurrent writing is not supported for a single file.

  • FTP itself does not provide data types. FTP Writer writes data of String type to FTP file.

Parameter description

  • datasource

    • Description: Data source name. It must be identical to the data source name added. Adding data source is supported in script mode.

    • Required: Yes

    • Default value: None

  • timeout

    • Description: The timeout threshold for connection to the FTP server (in milliseconds).

    • Required: No

    • Default value: 60,000 (1 min)

  • path

    • Description: The path of the FTP file system. FTP Writer writes multiple files under the path.

    • Required: Yes

    • Default value: None

  • fileName

    • Description: The name of the file written by FTP Writer. A random suffix is appended to the file name to form the actual name of the file written with each thread.

    • Required: Yes

    • Default value: None

  • writeMode

    • Description: The mode in which FTP Writer clears existing data before writing data:

      • truncate: Clear all the files with a prefix of fileName before writing.
      • append: No processing is done before writing, and Data Integration FTP Writer writes data directly using fileName without conflict of file names.
      • nonConflict: An error is reported if a file with a prefix of fileName exists under the path.
    • Required: Yes

    • Default value: None

  • fieldDelimiter

    • Description: The delimiter used to separate the written fields

    • Required: Yes (A single character)

    • Default value: None

  • compress

    • Description: gzip and bzip2 compression types are supported.

    • Required: No

    • Default value: No compression

  • encoding

    • Description: Encoding of the written files.

    • Required: No

    • Default value: utf-8

  • nullFormat

    • Description: Defining null (null pointer) with a standard string is not allowed in text files. Data Integration provides nullFormat to define which strings can be expressed as null.

      For example, when nullFormat="null" is configured, if source data is null, it is considered as a null field in Data Integration.

    • Required: No

    • Default value: None

  • dateFormat

    • Description: The format in which data of Date type is serialized into file, for example, “dateFormat”: “yyyy-MM-dd”.

    • Required: No

    • Default value: None

  • fileFormat

    • Description: The format in which a file is written, including text format and strict csv format. If the data to be written contains column delimiters, the column delimiters are escaped to double quotation marks in csv escape syntax. For text format, the data to be written is separated by column delimiters without being escaped.

    • Required: No

    • Default value: text

  • header

    • Description: The header used when a txt file is written. For example: [‘id’, ‘name’, ‘age’]

    • Required: No

    • Default value: None
  • markDoneFileName

    • Description: The name of the file marked as “done”. After a synchronization task is completed, a MarkDoneFile is generated, based on which whether the task is executed successfully is determined.

    • Required: No

    • Default value: None

Development in wizard mode

Currently, development in wizard mode is not supported.

Development in script mode

The following is a script configuration sample. For relevant parameters, see Parameter description.

  1. {
  2. "type": "job",
  3. "version": "1.0",
  4. "configuration": {
  5. "settting": {
  6. "key": "value"
  7. },
  8. "reader": {},
  9. "writer": {
  10. "plugin": "ftp",
  11. "parameter": {
  12. "datasource": "datasourceName",
  13. "timeout": "60000",
  14. "path": "/tmp/data/",
  15. "fileName": "yixiao",
  16. "writeMode": "truncate|append|nonConflict",
  17. "fieldDelimiter": ",",
  18. "encoding": "UTF-8",
  19. "nullFormat": "null",
  20. "dateFormat": "yyyy-MM-dd",
  21. "fileFormat": "csv",
  22. "header": []
  23. }
  24. }
  25. }
  26. }