Overview of OSS data synchronization capabilities in DataWorks - DataWorks

The OSS data source provides read and write access to OSS. This topic describes how DataWorks supports OSS data synchronization.

Supported field types and limitations

Batch data read

The OSS Reader reads data from OSS and converts it into a format that Data Integration can process. As OSS is an unstructured data storage service, the reader supports the following features.

Supported

Not supported

Supports TXT files, which must have a two-dimensional table schema.
Supports CSV-like files with a custom delimiter.

Note
Text formats (TXT and CSV) support gzip, bzip2, and zip compression.

You cannot package multiple files into a single compressed archive.
Supports files in ORC, Parquet, JSON, and JSONL formats.
Supports reading various data types (represented as STRING), column pruning, and column constants.
Supports recursive reads and object name filtering.
Supports concurrent reads from multiple objects.

The reader cannot read a single object (file) concurrently using multiple threads.
A single compressed object does not support concurrent multi-threaded reads.

Important

When you prepare data in OSS, CSV files must conform to the standard CSV format. For example, you must escape any double quotation mark (") in a column by replacing it with two double quotation marks (""), otherwise the file may be incorrectly split. If a file contains multiple delimiters, we recommend that you use the text type.
OSS is an unstructured data source that stores file-type data. Before you synchronize data, verify that the field structure of the data to be synchronized meets your expectations. Similarly, if the data structure of an unstructured data source changes, you must re-confirm the field structure in the task configuration. Otherwise, data synchronization errors may occur.

Batch data write

The OSS Writer converts data from the Data Integration protocol into text files in OSS. As OSS is an unstructured data storage service, the writer supports the following features.

Supported

Not supported

Only text files can be written. BLOB data such as videos and images is not supported. The text files must have a two-dimensional table schema.
Supports CSV-like files with a custom delimiter.
Supports ORC, Parquet, JSON, and JSONL formats.

Note
The SNAPPY compression format is supported in script mode.
Supports multi-threaded writes, with each thread writing to a different sub-file.
Supports file rolling. When a file exceeds a specified size, the writer switches to a new file.

A single file does not support concurrent writes.
OSS does not provide native data types. The OSS Writer writes all data to OSS objects as the STRING type.
If the storage class of the OSS bucket is Cold Archive Storage, write operations are not supported.
A single object (file) cannot exceed 100 GB.

Category	Data Integration column type
Integer	LONG
String	STRING
Floating-point	DOUBLE
Boolean	BOOLEAN
Date and time	DATE

Real-time write

Supports real-time write capabilities.
Single-table real-time write to data lakes is supported: Hudi (0.12.x), Paimon, and Iceberg.

Create a data source

Before you develop a synchronization task in DataWorks, you must add the required data source to DataWorks by following the instructions in Data source configuration. You can view parameter descriptions in the DataWorks console to understand the meanings of the parameters when you add a data source.

Note

To create an OSS data source across Alibaba Cloud accounts, you must grant the required permissions to the corresponding accounts. For more information, see Use bucket policies to grant cross-account access to OSS.
If you use the RAM role-based authorization mode to configure an OSS data source, see RAM role-based authorization: Configure a data source for more information.
To create an OSS data source across regions, we recommend that you use the public endpoint. For more information, see Access domains and network connectivity.

Data synchronization task development

For information about the entry point for and the procedure of configuring a synchronization task, see the following configuration guides.

Single-table batch synchronization task configuration guide

For the procedure, see Codeless UI configuration and Script mode configuration.
For all parameters and script demos available in script mode, see Appendix: script demos and parameter description below.

Single-table real-time synchronization task configuration guide

For the procedure, see Single-table real-time synchronization task configuration.

Full-database synchronization configuration guide

For the procedure, see Full-Database Batch Synchronization Task Capabilities and Full-Database Real-Time Synchronization Task Capabilities.

FAQ

Is there a file count limit when reading OSS files?

How do I handle dirty data when reading CSV files with multi-character delimiters?

Appendix: Script demos and parameter description

Configure a batch synchronization task by using the code editor

If you want to configure a batch synchronization task by using the code editor, you must configure the related parameters in the script based on the unified script format requirements. For more information, see Script mode configuration. The following information describes the parameters that you must configure for data sources when you configure a batch synchronization task by using the code editor.

Reader script demo: Generic example

The following example shows a generic script for OSS Reader.

{
    "type":"job",
    "version":"2.0",//The version number.
    "steps":[
        {
            "stepType":"oss",//The plug-in name.
            "parameter":{
                "nullFormat":"",//The string that represents a null value.
                "compress":"",//The text compression type.
                "datasource":"",//The data source.
                "column":[//The columns.
                    {
                        "index":0,//The column index.
                        "type":"string"//The data type.
                    },
                    {
                        "index":1,
                        "type":"long"
                    },
                    {
                        "index":2,
                        "type":"double"
                    },
                    {
                        "index":3,
                        "type":"boolean"
                    },
                    {
                        "format":"yyyy-MM-dd HH:mm:ss", //The time format.
                        "index":4,
                        "type":"date"
                    }
                ],
                "skipHeader":"",//CSV-like files may contain a header row. Set this parameter to skip the header.
                "encoding":"",//The encoding format.
                "fieldDelimiter":",",//The field delimiter.
                "fileFormat": "",//The text type.
                "object":[]//The object prefix.
            },
            "name":"Reader",
            "category":"reader"
        },
        {
            "stepType":"stream",
            "parameter":{},
            "name":"Writer",
            "category":"writer"
        }
    ],
    "setting":{
        "errorLimit":{
            "record":""//The maximum number of error records allowed.
        },
        "speed":{
            "throttle":true,//If throttle is set to false, the mbps parameter does not take effect, which means throttling is disabled. If throttle is set to true, throttling is enabled.
            "concurrent":1 //The number of concurrent threads.
            "mbps":"12",//The throttling rate. 1 mbps = 1 MB/s.
        }
    },
    "order":{
        "hops":[
            {
                "from":"Reader",
                "to":"Writer"
            }
        ]
    }
}

Reader script demo: Read ORC or Parquet files from OSS

OSS Reader reads ORC or Parquet files from OSS by reusing HDFS Reader. In addition to the existing OSS Reader parameters, the following extended parameters are added: Path (ORC) and FileFormat (ORC, Parquet) .

The following example shows how to read ORC files from OSS.

{
"stepType": "oss",
"parameter": {
"datasource": "",
"fileFormat": "orc",
"path": "/tests/case61/orc__691b6815_9260_4037_9899_****",
"column": [
{
"index": 0,
"type": "long"
},
{
"index": "1",
"type": "string"
},
{
"index": "2",
"type": "string"
}
]
}
}

The following example shows how to read Parquet files from OSS.

{
  "type":"job",
    "version":"2.0",
    "steps":[
    {
      "stepType":"oss",
      "parameter":{
        "nullFormat":"",
        "compress":"",
        "fileFormat":"parquet",
        "path":"/*",
        "parquetSchema":"message m { optional BINARY registration_dttm (UTF8); optional Int64 id; optional BINARY first_name (UTF8); optional BINARY last_name (UTF8); optional BINARY email (UTF8); optional BINARY gender (UTF8); optional BINARY ip_address (UTF8); optional BINARY cc (UTF8); optional BINARY country (UTF8); optional BINARY birthdate (UTF8); optional DOUBLE salary; optional BINARY title (UTF8); optional BINARY comments (UTF8); }",
        "column":[
          {
            "index":"0",
            "type":"string"
          },
          {
            "index":"1",
            "type":"long"
          },
          {
            "index":"2",
            "type":"string"
          },
          {
            "index":"3",
            "type":"string"
          },
          {
            "index":"4",
            "type":"string"
          },
          {
            "index":"5",
            "type":"string"
          },
          {
            "index":"6",
            "type":"string"
          },
          {
            "index":"7",
            "type":"string"
          },
          {
            "index":"8",
            "type":"string"
          },
          {
            "index":"9",
            "type":"string"
          },
          {
            "index":"10",
            "type":"double"
          },
          {
            "index":"11",
            "type":"string"
          },
          {
            "index":"12",
            "type":"string"
          }
        ],
        "skipHeader":"false",
        "encoding":"UTF-8",
        "fieldDelimiter":",",
        "fieldDelimiterOrigin":",",
        "datasource":"wpw_demotest_oss",
        "envType":0,
        "object":[
          "wpw_demo/userdata1.parquet"
        ]
      },
      "name":"Reader",
      "category":"reader"
    },
    {
      "stepType":"odps",
      "parameter":{
        "partition":"dt=${bizdate}",
        "truncate":true,
        "datasource":"0_odps_wpw_demotest",
        "envType":0,
        "column":[
          "id"
        ],
        "emptyAsNull":false,
        "table":"wpw_0827"
      },
      "name":"Writer",
      "category":"writer"
    }
  ],
    "setting":{
    "errorLimit":{
      "record":""
    },
    "locale":"zh_CN",
      "speed":{
      "throttle":false,
        "concurrent":2
    }
  },
  "order":{
    "hops":[
      {
        "from":"Reader",
        "to":"Writer"
      }
    ]
  }
}

Reader script demo: Read JSONLine files from OSS, parse the JSON data, and write the data to the destination

The following example shows how to read JSONLine data from OSS, parse the JSON data, vectorize specific fields in the JSON data, and then write the data to Milvus:

{
    "type": "job",
    "version": "2.0",
    "steps": [
        {
            "stepType": "oss",
            "parameter": {
                "datasource": "${OSS data source name}",
                "column": [
                    {
                        "name": "chunk_text",
                        "index": 0,
                        "type": "string"
                    }
                ],
                "fieldDelimiter": ",",
                "encoding": "UTF-8",
                "fileFormat": "jsonl",
                "object": [
                    "embedding/chunk1.jsonl"
                ]
            },
            "name": "Reader",
            "category": "reader"
        },
        {
            "stepType": "json-extracting",
            "parameter": {
                "column": [
                    {
                        "name": "text",
                        "fromColumn": "chunk_text",
                        "jsonPath": "$.text",
                        "type": "STRING",
                        "nullOrInvalidDataAction": "DIRTY_DATA"
                    }
                ]
            },
            "name": "jsonextract",
            "category": "flatmap"
        },
        {
            "stepType": "embedding-transformer",
            "parameter": {
                "modelProvider": "bailian",
                "modelName": "text-embedding-v4",
                "apiKey": "${the AccessKey that you want to use}",
                "embeddingColumns": {
                    "sourceColumnNames": [
                        "text"
                    ],
                    "embeddingColumnName": "my_vector"
                },
                "batchSize": 8,
                "dimension": 1024
            },
            "name": "embedding",
            "category": "flatmap"
        },
        {
            "stepType": "milvus",
            "parameter": {
                "schemaCreateMode": "ignore",
                "enableDynamicSchema": true,
                "datasource": "${Milvus data source name}",
                "column": [
                    {
                        "name": "my_vector",
                        "type": "FloatVector",
                        "dimension": "1024",
                        "elementType": "None",
                        "maxLength": "65535"
                    },
                    {
                        "name": "text",
                        "type": "VarChar",
                        "elementType": "None",
                        "maxLength": "65535"
                    }
                ],
                "collection": "yunshi_vector_07171130",
                "writeMode": "insert",
                "batchSize": 1024,
                "columnMapping": [
                    {
                        "sourceColName": "my_vector",
                        "dstColName": "my_vector"
                    },
                    {
                        "sourceColName": "text",
                        "dstColName": "text"
                    }
                ]
            },
            "name": "Writer",
            "category": "writer"
        }
    ],
    "setting": {
        "errorLimit": {
            "record": "0"
        },
        "speed": {
            "concurrent": 1
        }
    }
}

Reader script parameters

Parameter	Description	Required	Default value
datasource	The name of the data source. Script mode supports adding data sources. The value of this parameter must be the same as the name of the added data source.	Yes	N/A
Object	This parameter specifies one or more objects to synchronize from OSS. Three configuration methods are supported: explicit paths, wildcard paths, and dynamic parameter paths. 1. Configuration methods Explicit paths Basic rule: The path starts from the root directory of the bucket and does not need to include the bucket name. Specify a single file: Enter the full path of the file. Example: `my_folder/my_file.txt`. Specify multiple objects: Separate the paths of multiple files or folders with commas (`,`). Example: `folder_a/file1.txt`, `folder_a/file2.txt`. Wildcard paths You can use wildcards to match multiple files that meet a specific pattern. ``: Matches zero or more arbitrary characters. `?`: Matches exactly one arbitrary character. Examples: `abc[0-9].txt` can match `abc0.txt`, `abc10,txt`, `abc_test_9.txt`, and more. `abc?.txt` can match `abc1.txt`, `abcX.txt`, and more. Dynamic parameter paths You can embed scheduling parameters in the path to enable automated synchronization. When a task runs, the parameters are replaced with their actual values. Example: If you set the path to `raw_data/${bizdate}/abc.txt`, the task dynamically synchronizes the folder that corresponds to the business date on a daily basis. For more information about scheduling parameters, see Scheduling parameter sources and expressions. Important Use wildcards with caution. Using wildcards (especially ``) triggers a traversal scan of OSS paths. When the number of files is large, this can consume a significant amount of memory and time, and may even cause tasks to fail due to out-of-memory errors. We recommend that you do not use broad wildcards in production environments. If you encounter this issue, split the files into different directories before synchronization. The data synchronization system treats all objects synchronized in a single job as a single data table. You must ensure that all objects conform to the same schema. 2. Concurrent read mechanism and performance* The configuration method directly determines the concurrency performance of data extraction: Single-thread mode: When you specify only one explicit, non-compressed file, the task extracts data in single-thread mode. Multi-thread mode: When you specify multiple explicit files or use wildcards that match multiple files, the task automatically enables multi-threaded concurrent reads to significantly improve extraction efficiency. You can configure the number of concurrent threads in Channel Control.	Yes	None
parquetSchema	This parameter is used when you read OSS data in Parquet file format. It takes effect only when fileFormat is set to parquet, and specifies the type definition of the Parquet storage. Make sure that the overall configuration conforms to JSON syntax after you specify parquetSchema. `message MessageTypeName { required_or_optional, data_type, column_name; ......................; }` The configuration format of parquetSchema is described as follows: MessageType name: Specify a name. Required or optional: required indicates that the field cannot be null, and optional indicates that the field can be null. We recommend that you set all fields to optional. Data type: Parquet files support BOOLEAN, Int32, Int64, Int96, FLOAT, DOUBLE, BINARY (use BINARY for string types), and fixed_len_byte_array types. Each column definition must end with a semicolon, including the last line. The following example shows a sample configuration. `"parquetSchema": "message m { optional int32 minute_id; optional int32 dsp_id; optional int32 adx_pid; optional int64 req; optional int64 res; optional int64 suc; optional int64 imp; optional double revenue; }"`	No	None
column	The list of fields to read. type specifies the data type of the source data, index specifies the column number (zero-based) in the text file that the current column corresponds to, and value specifies that the current column is a constant. Instead of reading data from the source file, the system automatically generates the column based on the specified value. By default, you can read all data as the STRING type. The configuration is as follows. `"column": [""]` You can specify column field information. The configuration is as follows. `"column": { "type": "long", "index": 0 //Retrieve the INT field from the first column of the OSS text. }, { "type": "string", "value": "alibaba" //Generate the string field "alibaba" from OSSReader as the current field. }` Note* For the column information you specify, the type parameter is required, and you must specify either index or value.	Yes	All data is read as the STRING type.
fileFormat	The file type. The file type of the source OSS objects. Valid values: csv, text, parquet, orc, json, and jsonl. Both csv and text formats support custom delimiters.	Yes	csv
fieldDelimiter	The field delimiter for reading data. Note When the OSS Reader reads data, you must specify a field delimiter. If you do not specify one, a comma (,) is used by default. The comma (,) is also the default value in the console configuration. If the delimiter is invisible, enter its Unicode encoding. For example, \u001b or \u007c.	Yes	,
lineDelimiter	The line delimiter for reading data. Note This parameter takes effect only when fileFormat is set to text.	No	N/A
compress	The compression type for text files. By default, this parameter is left empty, which indicates that the data is not compressed. Valid values: gzip, bzip2, and zip.	No	No compression
encoding	The encoding used to read files.	No	utf-8
nullFormat	Standard strings cannot be used to define null (null pointer) in text files. Data Integration provides the nullFormat parameter to define which strings can represent null. Examples: If you configure `nullFormat:"null"`, which is equivalent to a "visible character", and the source data is null, Data Integration treats the field as a null field. If you configure `nullFormat:"\u0001"`, which is equivalent to an "invisible character", and the source data is the string "\u0001", Data Integration treats the field as a null field. If you do not specify the `"nullFormat"` parameter, which is equivalent to "not configured", the source data is written to the destination as-is without any conversion.	No	None
skipHeader	CSV-like files may contain a header row with column titles that needs to be skipped. By default, the header is not skipped. skipHeader is not supported in compressed file mode.	No	false
csvReaderConfig	The configuration for reading CSV-like files. This parameter is of the MAP type. The CsvReader is used to read CSV-like files and supports various configurations. If this parameter is not specified, the default values are used.	No	None
successOnNoObject	Specifies whether the task ends as successful when no objects exist in the specified read path. If this parameter is set to false, the task fails. If this parameter is set to true, the task ends as successful without writing any data. This parameter is applicable to periodic synchronization scenarios where the source path may be empty. In codeless UI mode, this parameter corresponds to "Allow Missing Source Files" and is displayed only when fileFormat is set to csv or text.	No	false
minLastModified	Specifies that only files whose last modified time is equal to or later than the specified time are synchronized. The format is `yyyyMMddHHmmss`. For example, if you configure `"minLastModified": "20250101000000"`, only files modified on or after January 1, 2025, 00:00:00 are read, and earlier files are filtered out. You can use this parameter together with `maxLastModified` to form a `[min, max)` left-closed, right-open interval.	No	N/A
maxLastModified	Synchronizes only files whose last modified time is earlier than the specified time. The format is `yyyyMMddHHmmss`. For example, if you set `"maxLastModified": "20250102000000"`, only files modified before 00:00:00 on January 2, 2025 are read. Files modified at exactly that time are excluded (right-open interval). You can use this parameter together with `minLastModified` to define a `[min, max)` left-closed, right-open interval.	No	N/A

Writer script demo: General example

The following is a general script example for OSS Writer.

{
    "type":"job",
    "version":"2.0",
    "steps":[
        {
            "stepType":"stream",
            "parameter":{},
            "name":"Reader",
            "category":"reader"
        },
        {
            "stepType":"oss",// The plug-in name.
            "parameter":{
                "nullFormat":"",// The null format provided by Data Integration, which defines what strings can represent null.
                "dateFormat":"",// The date format.
                "datasource":"",// The data source.
                "writeMode":"",// The write mode.
                "writeSingleObject":"false", // Specifies whether to write synchronized data to a single OSS file.
                "encoding":"",// The encoding format.
                "fieldDelimiter":","// The field delimiter.
                "fileFormat":"",// The file format.
                "object":""// The object prefix.
            },
            "name":"Writer",
            "category":"writer"
        }
    ],
    "setting":{
        "errorLimit":{
            "record":"0"// The maximum number of error records allowed.
        },
        "speed":{
            "throttle":true,// If throttle is set to false, the mbps parameter does not take effect and bandwidth is not throttled. If throttle is set to true, bandwidth is throttled.
            "concurrent":1, // The number of concurrent threads.
            "mbps":"12"// The throttling rate. 1 mbps = 1 MB/s.
        }
    },
    "order":{
        "hops":[
            {
                "from":"Reader",
                "to":"Writer"
            }
        ]
    }
}

Writer script demo: Writing ORC or Parquet files to OSS

OSS supports writing ORC or Parquet files by reusing HDFS Writer. In addition to the existing OSS Writer parameters, extended configuration parameters such as Path and FileFormat are added. For information about these parameters, see HDFS Writer.

The following is an example of writing ORC or Parquet files to OSS:

Important

The following is only an example. Modify the parameters based on your actual column names and types. Do not copy and use the example directly.

Write ORC files to OSS

To write ORC files, only the script mode is supported. You must switch to the script mode for configuration. Set fileFormat to orc, set path to the path of the file to write, and configure column in the format of {"name":"your column name","type": "your column type"}.

The following ORC types are supported for writing:

Field type	Batch write to OSS (ORC format)
TINYINT	Supported
SMALLINT	Supported
INT	Supported
BIGINT	Supported
FLOAT	Supported
DOUBLE	Supported
TIMESTAMP	Supported
DATE	Supported
VARCHAR	Supported
STRING	Supported
CHAR	Supported
BOOLEAN	Supported
DECIMAL	Supported
BINARY	Supported

{
"stepType": "oss",
"parameter": {
"datasource": "",
"fileFormat": "orc",
"path": "/tests/case61",
"fileName": "orc",
"writeMode": "append",
"column": [
{
"name": "col1",
"type": "BIGINT"
},
{
"name": "col2",
"type": "DOUBLE"
},
{
"name": "col3",
"type": "STRING"
}
],
"writeMode": "append",
"fieldDelimiter": "\t",
"compress": "NONE",
"encoding": "UTF-8"
}
}

Write to OSS in Parquet format

{
"stepType": "oss",
"parameter": {
"datasource": "",
"fileFormat": "parquet",
"path": "/tests/case61",
"fileName": "test",
"writeMode": "append",
"fieldDelimiter": "\t",
"compress": "SNAPPY",
"encoding": "UTF-8",
"parquetSchema": "message test { required int64 int64_col;\n required binary str_col (UTF8);\nrequired group params (MAP) {\nrepeated group key_value {\nrequired binary key (UTF8);\nrequired binary value (UTF8);\n}\n}\nrequired group params_arr (LIST) {\nrepeated group list {\nrequired binary element (UTF8);\n}\n}\nrequired group params_struct {\nrequired int64 id;\n required binary name (UTF8);\n }\nrequired group params_arr_complex (LIST) {\nrepeated group list {\nrequired group element {\n required int64 id;\n required binary name (UTF8);\n}\n}\n}\nrequired group params_complex (MAP) {\nrepeated group key_value {\nrequired binary key (UTF8);\nrequired group value {\nrequired int64 id;\n required binary name (UTF8);\n}\n}\n}\nrequired group params_struct_complex {\nrequired int64 id;\n required group detail {\nrequired int64 id;\n required binary name (UTF8);\n}\n}\n}",
"dataxParquetMode": "fields"
}
}

Writer script parameters

Parameter	Description	Required	Default value
datasource	The name of the data source. You can add a data source in script mode. The value of this parameter must be the same as the name of the data source that you add.	Yes	N/A
object	The name of the file that OSS Writer writes to. OSS uses file names to simulate a directory structure. OSS imposes the following limits on object names: If you use `"object": "datax"`, the written object name starts with datax, followed by a random string suffix. If you use `"object": "cdo/datax"`, the written object name starts with `/cdo/datax`, followed by a random string suffix. OSS uses a forward slash (/) as the directory separator. If you do not want a random UUID suffix, we recommend that you set `"writeSingleObject" : "true"`. For more information, see the description of writeSingleObject.	Yes	N/A
ossBlockSize	The block size for OSS multipart uploads. The default block size is 16 MB. When the file format is parquet or ORC, you can configure this parameter at the same level as the object parameter. OSS multipart uploads support a maximum of 10,000 parts. With the default block size, the maximum file size is 160 GB. If the number of parts exceeds the limit, you can increase the block size to support larger file uploads.	No	16
writeMode	Specifies how OSS Writer handles existing data before writing: truncate: Before writing, clears all objects whose names match the specified prefix. For example, if you set `"object":"abc"`, all objects whose names start with abc are cleared. append: No processing is performed before writing. Data Integration OSS Writer writes data directly using the object name and appends a random UUID suffix to ensure unique file names. For example, if you specify the object name as DI, the actual written object name is DI_**__**. nonConflict: If any object with a matching prefix exists at the specified path, an error is returned. For example, if you set `"object":"abc"` and an object named abc123 exists, an error is returned.	Yes	None
writeSingleObject	Specifies whether to write data to a single file when writing data to OSS: true: Write data to a single file. If no data is read, no empty file is generated. false: Write data to multiple files. If no data is read and a file header is configured, an empty file that contains only the file header is generated. Otherwise, only an empty file is generated. Note When you write data in ORC or Parquet format, the writeSingleObject parameter does not take effect. This means that you cannot use this parameter to write data to a single ORC or Parquet file in multi-concurrency scenarios. To write data to a single file, you can set the concurrency to 1. However, a random suffix is appended to the file name, and setting the concurrency to 1 reduces the synchronization speed. In certain scenarios, such as when the source is Hologres, data is read by shard. A single concurrency may still generate multiple files.	No	false
fileFormat	The format of the output file. The following formats are supported: csv: Only strict csv format is supported. If the data to be written contains column delimiters, the data is escaped based on the csv escape syntax. The escape character is a double quotation mark ("). text: The data to be written is simply separated by column delimiters. If the data to be written contains column delimiters, no escaping is performed. parquet: If you use this file type, you must add the parquetSchema parameter to define data types. Important To write data in Parquet format, you must switch to script mode and configure parquetSchema. For configuration examples, see Appendix: script demos and parameter description. If you do not configure parquetSchema, DataWorks automatically converts data types based on the source field types according to a specific strategy. For more information about the conversion strategy, see Appendix: conversion strategy for Parquet data types. ORC: If you use this format, you must switch to the script mode. json, jsonl: Data is written in the JSON format.	No	text
compress	The compression format of the data files written to OSS. You must configure this parameter in the script mode. Important Compression is not supported for CSV and TEXT formats. Parquet and ORC files support only SNAPPY compression.	No	None
fieldDelimiter	The field delimiter for writing.	No	,
encoding	The encoding configuration for the output file.	No	utf-8
parquetSchema	Required when writing to OSS in Parquet file format. This parameter describes the schema of the destination file and takes effect only when fileFormat is set to parquet. The format is as follows. `message MessageTypeName { required_or_optional, data_type, column_name; ......................; }` The configuration items are described as follows: MessageType name: Enter a name. Required or not: required indicates that the field cannot be null, and optional indicates that the field can be null. We recommend that you set all fields to optional. Data type: Parquet files support the following types: BOOLEAN, INT32, INT64, INT96, FLOAT, DOUBLE, BINARY (use BINARY for string types), and FIXED_LEN_BYTE_ARRAY. Note Each column definition must end with a semicolon, including the last line. Example: `message m { optional int64 id; optional int64 date_id; optional binary datetimestring; optional int32 dspId; optional int32 advertiserId; optional int32 status; optional int64 bidding_req_num; optional int64 imp; optional int64 click_num; }`	No	None
nullFormat	Standard strings cannot be used to define null (null pointer) in text files. The data synchronization system provides nullFormat to define a string that represents null. For example, if you set `nullFormat="null"` and the source data is `null`, the data synchronization system treats it as a null field.	No	None
header	The header row when writing to OSS. Example: `["id", "name", "age"]`.	No	None
maxFileSize (advanced configuration, not supported in codeless UI)	The maximum size of a single object file written to OSS. The default value is 10,000 × 10 MB. This is similar to controlling log file sizes when printing log4j logs. During OSS multipart upload, each part is 10 MB (which is also the minimum granularity for log rotation, meaning any maxFileSize value less than 10 MB is treated as 10 MB). Each OSS InitiateMultipartUploadRequest supports a maximum of 10,000 parts. When rotation occurs, the object name is generated by appending suffixes such as _1,_2,_3 to the original object prefix with a UUID-based random string. Note The default unit is MB. Configuration example: "maxFileSize":300 sets the maximum size of a single file to 300 MB. `maxFileSize` takes effect only for CSV and TEXT formats. The file size is calculated at the memory level of the synchronization task process and cannot precisely control the actual size of files written to the destination. During the write process, data expansion may cause the actual file size to exceed the expected value.	No	100,000
suffix (advanced configuration, not supported in codeless UI)	The suffix appended to the file name generated during data synchronization write. For example, if you set suffix to .csv, the final file name is fileName****.csv.	No	None

Appendix: Type conversion strategies for Parquet data

If you do not configure parquetSchema, DataWorks converts source field types to Parquet data types based on the following strategies.

Converted data type	Parquet type	Parquet logical type
CHAR / VARCHAR / STRING	BINARY	UTF8
BOOLEAN	BOOLEAN	N/A
BINARY / VARBINARY	BINARY	N/A
DECIMAL	FIXED_LEN_BYTE_ARRAY	DECIMAL
TINYINT	INT32	INT_8
SMALLINT	INT32	INT_16
INT/INTEGER	INT32	N/A
BIGINT	INT64	N/A
FLOAT	FLOAT	N/A
DOUBLE	DOUBLE	N/A
DATE	INT32	DATE
TIME	INT32	TIME_MILLIS
TIMESTAMP/DATETIME	INT96	N/A