DataWorks:Doris

Parameter

Description

Required

Default value

datasource

The name of the data source. It must be the same as the name of the added data source. You can add data sources by using the code editor.

Yes

No default value

table

The name of the table from which you want to read data. Each synchronization task can be used to synchronize data from only one table.

For a sharded table, you can use the table parameter to specify the partitions from which you want to read data. Examples:

• You can configure a range to read data from sharded databases and tables. For example, 'table_[0-99]' specifies that data is read from 'table_0', 'table_1', and 'table_2', up to 'table_99'.

• If your tables have numeric suffixes of the same length, such as 'table_000', 'table_001', and 'table_002', up to 'table_999', you can configure the parameter as '"table": ["table_00[0-9]", "table_0[10-99]", "table_[100-999]"]'.

Yes

No default value

column

The columns that you want to synchronize. The columns are described in a JSON array. By default, all columns are synchronized. For example, use ["*"].

• Column pruning: You can select a subset of columns to export.

• The column order can be changed. This indicates that you can specify columns in an order different from the order specified by the schema of the source table.

• Constant configuration: You must follow the Doris SQL syntax. Example: ["id","`table`","1","'mingya.wmy'","'null'","to_char(a+1)","2.3","true"].

  • id: a column name.

  • table: the name of a column that contains reserved keywords.

  • 1: an integer constant.

  • 'mingya.wmy': a string constant, which is enclosed in single quotation marks (').

  • null:

    • " " indicates an empty string.

    • null indicates a null value.

    • 'null' indicates the string "null".

  • to_char(a+1): a function expression that is used to calculate the length of a string.

  • 2.3: a floating-point constant.

  • true: a Boolean value.

• The column parameter must explicitly specify all the columns from which you want to read data. This parameter cannot be left empty.

Yes

No default value

splitPk

To improve read performance, you can use the splitPk parameter to specify a shard key. Data Integration uses this key to partition the data and run concurrent tasks.

• We recommend that you set the splitPk parameter to the name of the primary key column of the table. Data can be evenly distributed to different shards based on the primary key column, instead of being intensively distributed only to specific shards.

• The splitPk parameter supports sharding for data only of integer data types. If you set the splitPk parameter to a field of an unsupported data type, such as a string, floating point, or date data type, the setting of this parameter is ignored, and a single thread is used to read data.

• If the splitPk parameter is not provided or is left empty, a single thread is used to read data.

No

No default value

where

The filter condition. In many business scenarios, you may want to synchronize only the data from the current day. To do this, you can specify the where condition as gmt_create > $bizdate.

• You can use the WHERE clause to read incremental data. If the where parameter is not provided or is left empty, Doris Reader reads all data.

• Do not set the where parameter to limit 10. This value does not conform to the constraints of Doris on the SQL WHERE clause.

No

No default value

querySql (advanced parameter, which is available only in the code editor)

In some business scenarios, the where parameter may not be sufficient to describe the desired filter conditions. You can use this parameter to specify a custom SQL query for filtering. If you configure this parameter, Data Integration ignores the table, column, where, and splitPk parameters, and uses the custom query to retrieve data. For example, to join multiple tables before synchronization, you can use a query like select a,b from table_a join table_b on table_a.id = table_b.id. The querySql parameter has a higher priority than the table, column, where, and splitPk parameters. The datasource parameter is still required for the system to retrieve connection details such as the username and password.

No

No default value

Parameter

Description

Required

Default value

datasource

The name of the data source. It must be the same as the name of the added data source. You can add data sources by using the code editor.

Yes

No default value

table

The name of the table from which you want to read data.

Yes

No default value

column

The destination columns to write data to. Specify column names in an array, for example, "column":["id","name","age"]. To write data to all columns in order, use an asterisk (*) in the array, for example, "column":["*"].

Yes

No default value

preSql

The SQL statements to execute before the data synchronization task starts. In the codeless UI, you can execute only one SQL statement. In the code editor, you can execute multiple SQL statements. For example, you can use these statements to clear existing data from the table.

No

No default value

postSql

The SQL statement that you want to execute after the synchronization task is run. For example, you can set this parameter to the SQL statement that is used to add a timestamp. You can execute only one SQL statement on the codeless UI and multiple SQL statements in the code editor.

No

No default value

maxBatchRows

The maximum number of rows that you can write to the destination table at a time. Both this parameter and the batchSize parameter determine the number of data records that you can write to the destination table at a time. Each time the cached data reaches the value of either parameter, the writer starts to write the data to the destination table.

No

500000

batchSize

The maximum amount of data that you can write to the destination table at a time. Both this parameter and the maxBatchRows parameter determine the number of data records that you can write to the destination table at a time. Each time the cached data reaches the value of either parameter, the writer starts to write the data to the destination table.

No

104857600

maxRetries

The maximum number of retries allowed after you failed to write multiple data records to the destination table at a time.

No

3

labelPrefix

The label prefix for each uploaded file batch. The final label is a combination of labelPrefix + UUID to ensure that each import is idempotent, preventing data duplication.

No

datax_doris_writer_

loadProps

The request parameters for StreamLoad, mainly used to configure the import data format. By default, data is imported in CSV format. If the loadProps parameter is not configured, the default CSV format is used, with \t as the column delimiter and \n as the row delimiter. The default configuration is as follows:

"loadProps": {
    "format":"csv",
    "column_separator": "\t",
    "line_delimiter": "\n"
}

If you want to write data in the JSON format, use the following settings:

"loadProps": {
    "format": "json"
}

No

No default value