DataWorks:OSS data source

Support

Not supported

• Supports TXT files. The schema in the TXT file must be a two-dimensional table.

• Supports CSV-like files with custom delimiters.

  Note

  Text files (TXT and CSV) support gzip, bzip2, and zip compression.

  When you compress files, a compressed package cannot contain multiple files.

• Supports ORC and PARQUET formats.

• Supports reading multiple data types (represented as strings), column pruning, and column constants.

• Supports recursive reads and file name filtering.

• Supports concurrent reads for multiple objects.

• Does not support multi-threaded concurrent reads for a single object (file).

• Does not support multi-threaded concurrent reads for a single compressed object.

Support

Not supported

• Supports writing only text files (not BLOBs such as videos and images). The schema in the text file must be a two-dimensional table.

• Supports CSV-like files with custom delimiters.

• Supports ORC and PARQUET formats.

  Note

  The SNAPPY compression format is supported in the code editor.

• Supports multi-threaded writes. Each thread writes data to a different sub-file.

• Supports file rollover. When a file exceeds a specified size, the system switches to a new file.

• Does not support concurrent writes to a single file.

• OSS does not provide data types. OSS Writer writes all data as the STRING type to OSS objects.

• Does not support writing data if the storage class of the OSS bucket is Cold Archive.

• A single object (file) cannot exceed 100 GB.

Type classification

Data integration column configuration type

Integer types

LONG

String types

STRING

Floating-point types

DOUBLE

Boolean types

BOOLEAN

Date and time types

DATE

Parameter

Description

Required

Default value

datasource

The name of the data source. The value of this parameter must be the same as the name of the data source that you add in the code editor.

Yes

None

Object

Specifies one or more objects to sync from OSS. You can specify the object using a full path, a path with wildcard characters, or a path with dynamic parameters.

1. Configuration methods

• Specify the path.

  • Basic rule: The path starts from the root directory of the bucket and does not need to include the bucket name.

  • To specify a single file, enter its full path. Example: my_folder/my_file.txt.

  • To specify multiple objects, separate their paths with commas (,). Example: folder_a/file1.txt, folder_a/file2.txt.

• Path with wildcard characters

  • Use wildcard characters to match multiple files that follow a specific pattern.

  • *: Matches zero or more characters.

  • ?: Matches one character.

  • Examples:

    • abc*[0-9].txt matches abc0.txt, abc10,txt, abc_test_9.txt, and more.

    • abc?.txt matches abc1.txt, abcX.txt, and more.

• Path with dynamic parameters

  • Embed scheduling parameters in the path to automate synchronization. When the task runs, the parameters are replaced with their actual values.

  • Example: If you set the path to raw_data/${bizdate}/abc.txt, the task dynamically syncs the folder for the corresponding data timestamp each day.

  • For more information about scheduling parameters, see Sources and expressions of scheduling parameters.

2. Concurrent read mechanism and performance

The way you configure the path determines the concurrency and performance of data extraction:

• Single-threaded mode: The task runs in single-threaded mode if you specify a single, uncompressed file.

• Multi-threaded mode: The task automatically enables multi-threaded concurrent reads if you specify multiple files or use wildcard characters that match multiple files. This significantly improves extraction efficiency. You can configure the number of concurrent threads in the Channel Control section.

Yes

None

parquetSchema

This parameter is required only when you read Parquet files from OSS. It takes effect only when fileFormat is set to parquet. This parameter describes the data types in the Parquet file. Ensure that the configuration is in valid JSON format.

message MessageTypeName {
Required/Optional, Data type, Column name;
......................;
}

"parquetSchema": "message m { optional int32 minute_id; optional int32 dsp_id; optional int32 adx_pid; optional int64 req; optional int64 res; optional int64 suc; optional int64 imp; optional double revenue; }"

No

None

column

The list of fields to read. `type` specifies the data type of the source data. `index` specifies the column number (starting from 0) in the text file. `value` specifies that the current type is a constant. The data for this column is not read from the source file but is automatically generated based on the `value`.

"column": ["*"]

"column":
    {
       "type": "long",
       "index": 0    // Gets an int field from the first column of the OSS text file.
    },
    {
       "type": "string",
       "value": "alibaba"  // Generates a string field "alibaba" from within OSS Reader as the current field.
    }

Yes

All data is read as the STRING type.

fileFormat

The file format of the source object in OSS. Valid values are `csv` and `text`. Both formats support custom delimiters.

Yes

csv

fieldDelimiter

The column delimiter used to read the file.

Yes

,

lineDelimiter

The row delimiter.

No

None

compress

The compression format of the text file. The default value is empty, which means no compression. Supported formats are gzip, bzip2, and zip.

No

No compression

encoding

The encoding format of the source file.

No

utf-8

nullFormat

A text file cannot use a standard string to define a null pointer. Use `nullFormat` to define which strings represent null. For example:

• If you configure nullFormat:"null", data synchronization treats the string `null` in the source as a null field.

• If you set nullFormat:"\u0001", which is an invisible character, and the source data is the string "\u0001", the data is treated as a null field.

• If you do not set the "nullFormat" parameter, the source data is written to the destination as is, without any conversion.

No

None

skipHeader

Skips the header row in a CSV-like file. The default value is false. The skipHeader parameter is not supported for compressed files.

No

false

csvReaderConfig

The parameters for reading a CSV file. This is a map. The CsvReader is used to read CSV files. If you do not configure these parameters, default values are used.

No

None

Parameter

Description

Required

Default value

datasource

The name of the data source. The value of this parameter must be the same as the name of the data source that you add in the code editor.

Yes

None

object

The name of the file to be written to OSS. OSS uses file names to simulate a directory structure. OSS has the following limits on object names:

• If you set "object": "datax", the written object name starts with `datax` and has a random string appended.

• If you set "object": "cdo/datax", the written object name starts with /cdo/datax and has a random string appended. The forward slash (/) is used as a directory separator in OSS.

If you do not want a random UUID appended, set "writeSingleObject" : "true". For more information, see the description of writeSingleObject.

Yes

None

ossBlockSize

The size of each data block in MB. The default value is 16. This parameter is supported only when the `fileFormat` is parquet or ORC. You can configure this parameter at the same level as the object parameter.

Because multipart upload in OSS supports a maximum of 10,000 blocks, the default single file size is limited to 160 GB. If the number of blocks exceeds the limit, you can increase the block size to support larger file uploads.

No

16

writeMode

Specifies how to handle existing data before writing:

• truncate: Clears all objects that match the object name prefix before writing. For example, if you set "object":"abc", all objects starting with `abc` are cleared.

• append: No processing is performed before writing. OSS Writer directly writes data using the object name and appends a random UUID to ensure that file names do not conflict. For example, if you specify the object name as `DataIntegration`, the actual written name is DI_****_****_****.

• nonConflict: If an object with a matching prefix exists in the specified path, an error is reported. For example, if you set "object":"abc" and an object named `abc123` exists, an error is reported.

Yes

None

writeSingleObject

Specifies whether to write data to a single file:

• true: Writes data to a single file. If no data is read, no empty file is created.

• false: Writes data to multiple files. If no data is read and a file header is configured, an empty file containing only the header is created. Otherwise, only an empty file is created.

No

false

fileFormat

The format of the object file. The following formats are supported:

• csv: Only strict csv format is supported. If the data to be written includes a column delimiter, it will be escaped according to the csv escape syntax. The escape character is a double quotation mark (").

• text: Simply splits the data to be written using the column delimiter. No escaping is performed if the data includes the column delimiter.

• parquet: If you use this file type, you must add the `parquetSchema` parameter to define the data types.

  Important

  • To write data in Parquet format, you must switch to the code editor and configure `parquetSchema`. For a configuration example, see Appendix: Script demos and parameter descriptions.

  • If you do not configure `parquetSchema`, DataWorks converts the data types based on the source field types according to a specific policy. For more information about the conversion policy, see Appendix: Conversion policy for Parquet data types.

• ORC: If you use this format, you must switch to the code editor.

No

text

compress

The compression format of the object file written to OSS. This parameter must be configured in the code editor.

No

None

fieldDelimiter

The column delimiter.

No

,

encoding

Configure the file encoding.

No

utf-8

parquetSchema

This parameter is required when you write data to a Parquet file in OSS. It describes the structure of the object file. This parameter takes effect only when fileFormat is set to parquet. The format is as follows.

message MessageTypeName {
Required/Optional, Data type, Column name;
......................;
}

The configuration items are as follows:

• MessageTypeName: Enter a name.

• Required/Optional: `required` indicates that the field cannot be null. `optional` indicates that the field can be null. We recommend that you set all fields to `optional`.

• Data type: Parquet files support BOOLEAN, INT32, INT64, INT96, FLOAT, DOUBLE, BINARY (use BINARY for string types), and FIXED_LEN_BYTE_ARRAY types.

The following is an example.

message m {
optional int64 id;
optional int64 date_id;
optional binary datetimestring;
optional int32 dspId;
optional int32 advertiserId;
optional int32 status;
optional int64 bidding_req_num;
optional int64 imp;
optional int64 click_num;
}

No

None

nullFormat

A text file cannot use a standard string to define a null pointer. Use nullFormat to define the string that represents null. For example, if you set nullFormat="null" and the source data is null, the data is treated as a null field.

No

None

header

The header of the object file. Example: ["id", "name", "age"].

No

None

maxFileSize (Advanced configuration. This parameter is not supported in the codeless UI.)

The maximum size of a single object file in MB. The default value is 10,000 × 10 MB. This is similar to controlling the size of log files in log4j. When using multipart upload in OSS, each block is 10 MB (which is also the minimum granularity for log file rotation, meaning a `maxFileSize` less than 10 MB is treated as 10 MB). Each OSS InitiateMultipartUploadRequest supports a maximum of 10,000 blocks.

When rotation occurs, the object name is formed by appending suffixes such as _1, _2, _3 to the original object prefix with a random UUID.

No

100,000

suffix (Advanced configuration. This parameter is not supported in the codeless UI.)

The suffix of the generated file name. For example, if you set suffix to .csv, the final file name will be fileName****.csv.

No

None