DataWorks:SelectDB data source

Parameter

Description

Required

Default value

datasource

The name of the data source. The code editor supports adding data sources. The value of this parameter must be the same as the name of the added data source.

Yes

None

table

The name of the table that you want to synchronize.

Yes

None

column

The columns in the target table to which you want to write data. Separate the column names with commas (,). For example, "column":["id","name","age"]. If you want to write data to all columns in order, use (*). For example, "column":["*"].

Yes

None

preSql

The SQL statement to be executed before the data synchronization task runs. The codeless UI supports only one SQL statement. The code editor supports multiple SQL statements. For example, you can clear old data from a table before the task runs.

No

None

postSql

The SQL statement to be executed after the data synchronization task runs. The codeless UI supports only one SQL statement. The code editor supports multiple SQL statements. For example, you can add a timestamp after the task runs.

No

None

maxBatchRows

The maximum number of rows in each data batch to be imported. This parameter and the batchSize parameter control the import size of each batch. When the data in a batch reaches either of the thresholds, the import of that batch starts.

No

500000

batchSize

The maximum data volume of each data batch to be imported. This parameter and the maxBatchRows parameter control the import size of each batch. When the data in a batch reaches either of the thresholds, the import of that batch starts.

No

94371840

maxRetries

The number of retries after a data batch fails to be imported.

No

3

labelPrefix

The label prefix for each uploaded file. The final label is composed of labelPrefix + UUID to form a globally unique label. This ensures that data is not repeatedly imported.

No

datax_selectdb_writer_

loadProps

The request parameters for COPY INTO. This parameter is mainly used to configure the format of the imported data. By default, data is imported in JSON format. If loadProps is not configured or is set to "loadProps":{}, the default JSON format is used, as shown in the following configuration. (Currently, SelectDB supports only strip_outer_array=true).

"loadProps": {
    "format": "json",
    "strip_outer_array":true
}

To import data in CSV format, you can specify the format as CSV and configure the row and column delimiters, as shown in the following example. If you do not specify the row and column delimiters, all imported data is converted to strings. The tab character (\t) is used as the column delimiter, and the newline character (\n) is used as the row delimiter. This data is then combined into a CSV file for import into SelectDB.

"loadProps": {
    "format":"csv",
    "column_separator": "\\x01",
    "line_delimiter": "\\x02"
}

No

None

clusterName

The name of the SelectDB Cloud cluster. You can view the name in the SelectDB console.

No

None

flushInterval

The time interval for writing data batches, in milliseconds. If the values of the maxBatchRows and batchSize parameters are large, the system may import the data based on the write interval before the specified data volume is reached.

No

30000