DataWorks:OSS data source

Supported

Unsupported

• Reads TXT files. The schema in the TXT file must be a two-dimensional table.

• Reads CSV-like files with custom delimiters.

• Reads ORC and PARQUET formats.

• Reads multiple data types that are represented as strings. Supports column pruning and column constants.

• Supports recursive reads and file name filtering.

• Supports text compression. The available compression formats are gzip, bzip2, and zip.

  Note

  A compressed package cannot contain multiple files.

• Supports concurrent reads for multiple objects.

• Does not support multi-threaded concurrent reads for a single object (file).

• A single compressed object cannot be read by multiple threads concurrently due to technical limitations.

Supported

Unsupported

• Writes only text files. BLOB files such as videos and images are not supported. The schema in the text files must be a two-dimensional table.

• Writes CSV-like files with custom delimiters.

• Writes ORC and PARQUET formats.

• Supports multi-threaded writes. Each thread writes to a different sub-file.

• Supports file rotation. A new file is created when the current file exceeds a specific size.

• Does not support concurrent writes to a single file.

• OSS does not provide data types. OSS Writer writes all data to OSS objects as the STRING type.

• Writing to OSS buckets that use the Cold Archive storage class is not supported.

• A single object (file) cannot exceed 100 GB.

Type category

Data Integration column configuration type

Integer

LONG

String

STRING

Floating-point

DOUBLE

Boolean

BOOLEAN

Date and time

DATE

Parameter

Description

Required

Default value

datasource

The name of the data source. You can add a data source in the code editor. The value of this parameter must be the same as the name of the added data source.

Yes

None

Object

Specifies one or more objects to synchronize from OSS. You can specify an object using an explicit path, a wildcard character, or a dynamic parameter.

1. Configuration methods

• Explicit path

  • Basic rule: The path starts from the root directory of the bucket and does not need to include the bucket name.

  • To specify a single file, enter the full path of the file. Example: my_folder/my_file.txt.

  • Specify multiple objects: Use a comma (,) to separate multiple file or folder paths. For example: folder_a/file1.txt, folder_a/file2.txt.

• Wildcard path

  • You can use a wildcard character to match multiple files that follow a specific pattern.

  • *: Matches zero or more characters.

  • ?: Matches a single character.

  • Examples:

    • abc*[0-9].txt matches abc0.txt, abc10,txt, abc_test_9.txt, and so on.

    • abc?.txt matches abc1.txt, abcX.txt, and so on.

• Dynamic parameter path

  • You can embed scheduling parameters in the path to automate synchronization. When the task runs, the parameters are replaced with their actual values.

  • Example: If you set the path to raw_data/${bizdate}/abc.txt, the task can dynamically synchronize the folder for the corresponding data timestamp each day.

  • For more information about how to use scheduling parameters, see Supported formats of scheduling parameters.

2. Concurrent read mechanism and performance

The configuration method directly affects the concurrent performance of data extraction:

• Single-threaded mode: When you specify only a single, explicit, uncompressed file, the task extracts data in single-threaded mode.

• Multi-threaded mode: When you specify multiple explicit files, or match multiple files with a wildcard character, the task automatically enables multi-threaded concurrent reads to significantly improve extraction efficiency. You can configure the number of concurrent threads in the Channel Control section.

Yes

None

parquetSchema

This parameter is configured when you read a Parquet file from OSS. It takes effect only when fileFormat is set to parquet. This parameter specifies the data types stored in the Parquet file. You must ensure that the overall configuration conforms to the JSON syntax after you specify this parameter.

message MessageTypeName {
Required/Optional, Data type, Column name;
......................;
}

"parquetSchema": "message m { optional int32 minute_id; optional int32 dsp_id; optional int32 adx_pid; optional int64 req; optional int64 res; optional int64 suc; optional int64 imp; optional double revenue; }"

No

None

column

The list of fields to read. `type` specifies the data type of the source data. `index` specifies the column from which to read data. The value of `index` starts from 0. `value` specifies that the current column is a constant. Data is not read from the source file but is automatically generated based on the value of this parameter.

"column": ["*"]

"column":
    {
       "type": "long",
       "index": 0    // Reads an integer field from the first column of the OSS text file.
    },
    {
       "type": "string",
       "value": "alibaba"  // Generates a string field with the value "alibaba" from within OSS Reader as the current field.
    }

Yes

All data is read as the STRING type.

fileFormat

The format of the source files in OSS. Valid values: csv and text. Both formats support custom delimiters.

Yes

csv

fieldDelimiter

The delimiter that separates columns in the source files.

Yes

,

lineDelimiter

The delimiter that separates rows in the source files.

No

None

compress

The compression format of the text files. The default value is empty, which indicates that the files are not compressed. Valid values: gzip, bzip2, and zip.

No

Not compressed

encoding

The encoding format of the source files.

No

utf-8

nullFormat

In a text file, you cannot use a standard string to define a null pointer. Data Integration provides the nullFormat parameter to define which strings can be interpreted as null. Examples:

• If you set nullFormat:"null", which is a visible character, and the source data is the string "null", Data Integration interprets it as a null field.

• If you set nullFormat:"\u0001", which is an invisible character, and the source data is the string "\u0001", Data Integration interprets it as a null field.

• If you do not specify the "nullFormat" parameter, the source data is written to the destination as is, without any conversion.

No

None

skipHeader

Specifies whether to skip the header in a CSV-like file. The default value is false. The skipHeader parameter is not supported for compressed files.

No

false

csvReaderConfig

The parameters for reading a CSV file. This parameter is of the Map type. CsvReader is used to read CSV files. You can configure multiple parameters. If you do not configure this parameter, the default values are used.

No

None

Parameter

Description

Required

Default value

datasource

The name of the data source. You can add a data source in the code editor. The value of this parameter must be the same as the name of the added data source.

Yes

None

object

The name of the file to be written by OSS Writer. OSS uses file names to simulate folders. OSS has the following limits on object names:

• If you set "object": "datax", the name of the written object starts with `datax` and is followed by a random string.

• If you set "object": "cdo/datax", the name of the written object starts with /cdo/datax and is followed by a random string. The forward slash (/) is used as a delimiter to simulate a folder in OSS.

If you do not want a random UUID as a suffix, set "writeSingleObject" : "true". For more information, see the description of the writeSingleObject parameter.

Yes

None

ossBlockSize

The size of an OSS part. The default size is 16 MB. This parameter is supported only when the file is written in parquet or ORC format. You can add this parameter at the same level as the object parameter.

Because OSS multipart upload supports a maximum of 10,000 parts, the default size of a single file is limited to 160 GB. If the number of parts exceeds the limit, you can increase the part size to support larger file uploads.

No

16

writeMode

The data processing method that OSS Writer uses before it writes data:

• truncate: Clears all objects that match the specified object name prefix before writing data. For example, if you set "object":"abc", all objects whose names start with `abc` are cleared.

• append: Performs no processing before writing data. OSS Writer directly writes data to an object with the specified name. A random UUID is used as a suffix to ensure that file names do not conflict. For example, if you specify the object name as `DI`, the actual object name is DI_****_****_****.

• nonConflict: Reports an error if an object that matches the prefix exists in the specified path. For example, if you set "object":"abc" and an object named `abc123` exists, an error is reported.

Yes

None

writeSingleObject

Specifies whether to write data to a single file in OSS:

• true: Writes data to a single file. When no data can be read, no empty file is generated.

• false: Writes data to multiple files. When no data can be read, if a file header is configured, an empty file containing only the file header is output. Otherwise, only an empty file is output.

No

false

fileFormat

The format of the file to be written. The following formats are supported:

• csv: Only the strict csv format is supported. If the data to be written includes a column delimiter, it is escaped according to the csv escape syntax. The escape character is a double quotation mark (").

• text: The data to be written is simply split by the column delimiter. If the data to be written includes a column delimiter, it is not escaped.

• parquet: If you use this file format, you must add the parquetSchema parameter to define the data types.

  Important

  • To write data in Parquet format, you must switch to the code editor and configure the parquetSchema parameter. For a configuration example, see Appendix: Script demo and parameter description.

  • If you do not configure the parquetSchema parameter, DataWorks converts the data types based on a specific policy for the source field types. For more information about the conversion policy, see Appendix: Conversion policy for Parquet data types.

• ORC: To use this format, you must switch to the code editor.

No

text

compress

The compression format of the data file to be written to OSS. This parameter must be configured in the code editor.

No

None

fieldDelimiter

The delimiter that separates columns in the destination file.

No

,

encoding

The encoding format of the destination file.

No

utf-8

parquetSchema

This parameter is required when you write data to OSS in Parquet format. It is used to describe the structure of the destination file. This parameter takes effect only when fileFormat is set to parquet. The format is as follows.

message MessageTypeName {
Required/Optional, Data type, Column name;
......................;
}

The following list describes the configuration items:

• MessageTypeName: Enter a name.

• Required/Optional: `required` indicates that the field cannot be null. `optional` indicates that the field can be null. We recommend that you set this to `optional` for all fields.

• Data type: Parquet files support the BOOLEAN, INT32, INT64, INT96, FLOAT, DOUBLE, BINARY (use BINARY for string types), and FIXED_LEN_BYTE_ARRAY types.

Consider the following example.

message m {
optional int64 id;
optional int64 date_id;
optional binary datetimestring;
optional int32 dspId;
optional int32 advertiserId;
optional int32 status;
optional int64 bidding_req_num;
optional int64 imp;
optional int64 click_num;
}

No

None

nullFormat

You cannot use a standard string to define null (null pointer) in a text file. The data synchronization system provides the nullFormat parameter to define a string that represents a null value. For example, if you configure nullFormat="null" and the source data is null, the data synchronization system treats it as a null field.

No

None

header

The header of the file written to OSS. Example: ["id", "name", "age"].

No

None

maxFileSize (Advanced configuration, not supported in the codeless UI)

The maximum size of a single object file written to OSS. The default value is 10,000 × 10 MB. This is similar to controlling the size of a log file when you print log4j logs. When OSS performs multipart uploads, the size of each part is 10 MB. This is also the minimum granularity for log file rotation. A maxFileSize value less than 10 MB is treated as 10 MB. Each OSS InitiateMultipartUploadRequest supports a maximum of 10,000 parts.

When rotation occurs, the object name is formed by adding a UUID and a suffix such as _1, _2, or _3 to the original object prefix.

No

100,000

suffix (Advanced configuration, not supported in the codeless UI)

The suffix of the file name generated when data is written. For example, if you set suffix to .csv, the final file name is fileName****.csv.

No

None