DataWorks:OSS data source

Support

Unsupported

• Supports reading TXT files. The schema in the TXT file must be a two-dimensional table.

• Supports CSV-like files with custom delimiters.

• Supports ORC and PARQUET formats.

• Supports reading multiple data types, which are represented as strings. Supports column pruning and column constants.

• Supports recursive reads and file name filtering.

• Supports text compression. The available compression formats are gzip, bzip2, and zip.

  Note

  A compressed package cannot contain multiple compressed files.

• Supports concurrent reads for multiple objects.

• Does not support multi-threaded concurrent reads for a single object or file.

• A single compressed object technically cannot be read by multiple threads concurrently.

Support

Unsupported

• Supports writing only text files. BLOB files such as videos and images are not supported. The schema in the text file must be a two-dimensional table.

• Supports CSV-like files with custom delimiters.

• Supports ORC and PARQUET formats.

• Supports multi-threaded writes. Each thread writes to a different sub-file.

• Supports file scrolling. When a file size exceeds a specific value, the system switches to a new file.

• Does not support concurrent writes to a single file.

• OSS itself does not provide data types. OSS Writer writes all data to OSS objects as the STRING type.

• Writing to OSS buckets with the Cold Archive storage class is not supported.

• A single object or file cannot exceed 100 GB.

Type classification

Data Integration column configuration type

Integer

LONG

String

STRING

Floating-point

DOUBLE

Boolean

BOOLEAN

Date and time

DATE

Parameter

Description

Required

Default value

datasource

The name of the data source. In the code editor, you can add a data source. The value of this parameter must be the same as the name of the added data source.

Yes

None

Object

Specifies one or more objects to synchronize from OSS. You can configure this parameter in three ways: explicit path, wildcard path, and dynamic parameter path.

1. Configuration methods

• Explicit path

  • Basic rule: The path starts from the root directory of the bucket and does not need to include the bucket name.

  • Specify a single file: Enter the full path of the file. Example: my_folder/my_file.txt.

  • Specify multiple objects: Use a comma (,) to separate the paths of multiple files or folders. Example: folder_a/file1.txt, folder_a/file2.txt.

• Wildcard path

  • Use wildcards to match multiple files that follow a specific pattern.

  • *: Matches zero or more of any character.

  • ?: Matches one of any character.

  • Examples:

    • abc*[0-9].txt matches abc0.txt, abc10.txt, abc_test_9.txt, and so on.

    • abc?.txt matches abc1.txt, abcX.txt, and so on.

• Dynamic parameter path

  • Embed scheduling parameters in the path to automate synchronization. When the task runs, the parameters are replaced with their actual values.

  • Example: If you set the path to raw_data/${bizdate}/abc.txt, the task can dynamically synchronize the folder for the corresponding data timestamp each day.

  • For more information about how to use scheduling parameters, see Sources and expressions of scheduling parameters.

2. Concurrent read mechanism and performance

The configuration method directly affects the concurrent performance of data extraction:

• Single-threaded mode: When you specify only a single, non-compressed file, the task extracts data in single-threaded mode.

• Multi-threaded mode: When you specify multiple specific files or use a wildcard character to match multiple files, the task automatically enables multi-threaded concurrent reading to significantly improve extraction efficiency. You can configure the concurrency in Channel Control.

Yes

None

parquetSchema

This parameter is configured when reading a Parquet file from OSS. It takes effect only when fileFormat is set to parquet. It specifies the type description of the Parquet storage. After you fill in the parquetSchema, ensure that the overall configuration conforms to JSON syntax.

message MessageTypeName {
Required/Optional, Data Type, Column Name;
......................;
}

"parquetSchema": "message m { optional int32 minute_id; optional int32 dsp_id; optional int32 adx_pid; optional int64 req; optional int64 res; optional int64 suc; optional int64 imp; optional double revenue; }"

No

None

column

The list of fields to read. `type` specifies the type of the source data. `index` specifies which column of the text the current column comes from (starting from 0). `value` specifies that the current type is a constant. The data is not read from the source file, but the corresponding column is automatically generated based on the `value`.

"column": ["*"]

"column":
    {
       "type": "long",
       "index": 0    // Get the int field from the first column of the OSS text.
    },
    {
       "type": "string",
       "value": "alibaba"  // Generate a string field "alibaba" from within OSS Reader as the current field.
    }

Yes

All data is read as the STRING type.

fileFormat

The text type. The file type of the source OSS file. For example, csv or text. Both formats support custom delimiters.

Yes

csv

fieldDelimiter

The column delimiter for reading.

Yes

,

lineDelimiter

The row delimiter for reading.

No

None

compress

The text compression type. By default, this is not filled in, which means no compression. Supported compression types are gzip, bzip2, and zip.

No

No compression

encoding

The encoding configuration for reading files.

No

utf-8

nullFormat

In text files, null (a null pointer) cannot be defined with a standard string. Data synchronization provides nullFormat to define which strings can represent null. For example:

• If you configure nullFormat:"null", which is a visible character, and the source data is "null", data synchronization treats it as a null field.

• If you configure nullFormat:"\u0001", which is an invisible character, and the source data is the string "\u0001", data synchronization treats it as a null field.

• If you do not include the "nullFormat" parameter, which means it is not configured, the source data is written to the destination as is, without any conversion.

No

None

skipHeader

CSV-like files may have a header row that needs to be skipped. By default, it is not skipped. skipHeader is not supported in compressed file mode.

No

false

csvReaderConfig

Parameter configuration for reading CSV files. This is a Map type. CSV files are read using CsvReader, which has many configurations. If not configured, default values are used.

No

None

Parameter

Description

Required

Default value

datasource

The name of the data source. In the code editor, you can add a data source. The value of this parameter must be the same as the name of the added data source.

Yes

None

object

The name of the file written by OSS Writer. OSS uses file names to simulate directories. OSS has the following restrictions on object names:

• If you use "object": "datax", the written object starts with "datax" and a random string is added as a suffix.

• If you use "object": "cdo/datax", the written object starts with /cdo/datax and a random string is added as a suffix. The delimiter for simulating directories in OSS is a forward slash (/).

If you do not need a random UUID suffix, we recommend that you configure "writeSingleObject" : "true". For more information, see the description of writeSingleObject.

Yes

None

ossBlockSize

The OSS block size. The default block size is 16 MB. When the file is written in parquet or ORC format, you can add this parameter at the same level as the object parameter.

Because OSS multipart upload supports a maximum of 10,000 blocks, the default single file size is limited to 160 GB. If the number of blocks exceeds the limit, you can increase the block size to support larger file uploads.

No

16

writeMode

The data processing method before OSS Writer writes data:

• truncate: Clears all objects that match the object name prefix before writing. For example, if "object":"abc", all objects starting with "abc" will be cleared.

• append: No processing is done before writing. Data Integration OSS Writer directly writes using the object name and adds a random UUID suffix to ensure that file names do not conflict. For example, if the object name you specify is "DI", the actual written name will be DI_****_****_****.

• nonConflict: If an object with a matching prefix is found at the specified path, an error is reported directly. For example, if "object":"abc" and an object named "abc123" exists, an error will be reported.

Yes

None

writeSingleObject

Specifies whether to write data to a single file when writing to OSS:

• true: Writes to a single file. When no data can be read, no empty file is generated.

• false: Writes to multiple files. When no data can be read, if a file header is configured, an empty file containing only the file header is output. Otherwise, only an empty file is output.

No

false

fileFormat

The format of the output file. The following formats are supported:

• csv: Only strict csv format is supported. If the data to be written includes a column delimiter, it will be escaped according to the csv escape syntax. The escape character is a double quotation mark (").

• text: Simply splits the data to be written using the column delimiter. No escaping is performed if the data to be written includes the column delimiter.

• parquet: If you use this file type, you must add the parquetSchema parameter to define the data type.

  Important

  • To write data in parquet format, you need to switch to the code editor and configure parquetSchema. For a configuration example, see Appendix: Script demos and parameter descriptions.

  • If you do not configure parquetSchema, DataWorks will convert the data types according to a certain policy based on the source field types. For more information about the conversion policy, see Appendix: Conversion policy for Parquet data types.

• ORC: If you use this format, you need to switch to the code editor.

No

text

compress

The compression format of the data file written to OSS. This must be configured in a script task.

No

None

fieldDelimiter

The column delimiter for writing.

No

,

encoding

The encoding configuration of the output file.

No

utf-8

parquetSchema

This is a required parameter for writing to OSS in Parquet file format. It describes the structure of the object file. Therefore, this parameter is effective only when fileFormat is set to parquet. The format is as follows.

message MessageTypeName {
Required/Optional, Data Type, Column Name;
......................;
}

The configuration items are described as follows:

• MessageType name: Enter a name.

• Required/Optional: `required` means not null, `optional` means nullable. We recommend that you set all to `optional`.

• Data type: Parquet files support BOOLEAN, INT32, INT64, INT96, FLOAT, DOUBLE, BINARY (use BINARY for string types), and FIXED_LEN_BYTE_ARRAY types.

The following is an example.

message m {
optional int64 id;
optional int64 date_id;
optional binary datetimestring;
optional int32 dspId;
optional int32 advertiserId;
optional int32 status;
optional int64 bidding_req_num;
optional int64 imp;
optional int64 click_num;
}

No

None

nullFormat

In text files, null (a null pointer) cannot be defined with a standard string. The data synchronization system provides nullFormat to define which strings can represent null. For example, if you configure nullFormat="null" and the source data is null, the data synchronization system will treat it as a null field.

No

None

header

The header of the output file when writing to OSS. For example, ["id", "name", "age"].

No

None

maxFileSize (Advanced configuration, not supported in codeless UI)

The maximum size of a single object file when writing to OSS. The default is 10,000 × 10 MB. This is similar to controlling the size of a log file when printing log4j logs. When uploading in parts to OSS, each part is 10 MB (which is also the minimum granularity for log rotation files, meaning a maxFileSize less than 10 MB will be treated as 10 MB). Each OSS InitiateMultipartUploadRequest supports a maximum of 10,000 parts.

When rotation occurs, the object name rule is to append suffixes like _1, _2, _3 to the original object prefix with a random UUID.

No

100,000

suffix (Advanced configuration, not supported in codeless UI)

The suffix of the file name generated when data synchronization writes data. For example, if you configure suffix as .csv, the final written file name will be fileName****.csv.

No

None