DataWorks:StarRocks data source

Parameter

Description

Required

Default value

datasource

The name of the StarRocks data source.

Yes

None

selectedDatabase

The name of the StarRocks database.

No

The database is the one that is configured in the StarRocks data source.

column

The names of the columns from which you want to read data.

Yes

None

where

The filter condition. In actual business scenarios, you often need to synchronize data of the current day. In this case, you can set the where condition to gmt_create>$bizdate.

• The where condition can effectively implement incremental business data synchronization.

• If you do not specify the where statement or do not provide a key or value for where, all data is synchronized.

No

None

table

The name of the table from which you want to read data.

Yes

None

splitPk

When StarRocks Reader extracts data, if you specify splitPk, it indicates that you want to use the field represented by splitPk for data sharding. The data synchronization will then start concurrent tasks for data synchronization, improving the efficiency of data synchronization. We recommend that you set the splitPk parameter to the name of the primary key column of the source table. This way, data can be evenly distributed to different shards based on the primary key column, instead of being intensively distributed only to specific shards.

No

None

Parameter

Description

Required

Default value

datasource

The name of the StarRocks data source.

Yes

None

selectedDatabase

The name of the StarRocks database.

No

The name of the database that is configured in the StarRocks data source.

loadProps

The request parameters for the StarRocks Stream Load import method. If you want to import data as CSV files by using the Stream Load import method, you can configure request parameters. If you have no special requirements, set the parameter to {}. Request parameters that you can configure for the Stream Load import method:

• column_separator: The column delimiter for CSV import. Default value: \t.

• row_delimiter: The row delimiter for CSV import. Default value: \n.

If the data you want to write to StarRocks contains \t or \n, you must use other characters as delimiters. Example:

{"column_separator":"\\x01","row_delimiter":"\\x02"}

Stream Load also supports importing data in JSON format. You can configure:

{
  "format": "json"
}

Parameters that can be configured in JSON format:

• strip_outer_array: Specifies whether to crop the outermost array structure. Valid values: true and false. Default value: false.

  In real business scenarios, the JSON data to be imported may have a pair of brackets [] representing an array structure at the outermost layer. In this case, we recommend that you set this parameter to true, so that StarRocks will crop the outer brackets [] and import each inner array as a separate row of data. If you set this parameter to false, StarRocks will parse the entire JSON data file as an array and import it as a single row of data.

  For example, if the JSON data to be imported is as follows:

  [{"category":1,"author":2},{"category":3,"author":4}]
  

  • If you set this parameter to true, StarRocks will parse {"category":1,"author":2} and {"category":3,"author":4} as two rows of data and import them into the corresponding data rows in the target StarRocks table.

  • If you set this parameter to false, StarRocks will parse the entire JSON array as a single row of data and import it into the target StarRocks table.

• ignore_json_size: Specifies whether to check the size of the JSON Body in the HTTP request.

  Note

  The size of the JSON Body in the HTTP request cannot exceed 100MB by default. If the size of the JSON Body exceeds 100MB, the error The size of this batch exceed the max size [104857600] of json type data data [8617627793].Set ignore_json_size to skip check,although it may lead enormous memory consuming. will be prompted. To avoid this error, you can add ignore_json_size: true to the HTTP request header to ignore the check on the size of the JSON Body.

• compression: Specifies which compression algorithm to use during the StreamLoad data transfer process. The supported algorithms are GZIP, BZIP2, LZ4_FRAME, and ZSTD.

Yes

None

column

The names of the columns to which you want to write data.

Yes

None

loadUrl

Enter the StarRocks FrontEnd IP and Http Port (usually default is 8030). If there are multiple FrontEnd nodes, you can configure all of them and separate them with commas (,).

Yes

None

table

The name of the table to which you want to write data.

Yes

None

preSql

The SQL statement that you want to execute before the synchronization task is run. For example, you can execute the TRUNCATE TABLE tablename statement to delete outdated data before the synchronization task is run.

No

None

postSql

The SQL statement that you want to execute after the data synchronization task is run.

No

None

maxBatchRows

The maximum number of rows of data written each time.

No

500000

maxBatchSize

The maximum number of bytes written each time.

No

5242880