DataWorks:Snowflake data source

Parameter

Description

Required

Default value

datasource

The name of the Snowflake data source.

Yes

None

table

The name of the table from which you want to sync data.

Yes

None

schema

The schema where the Snowflake table that you want to read resides.

Yes

None

column

The columns that you want to sync from the source table. Use a JSON array to describe the fields. By default, all columns are used, such as ["*"].

• You can select a subset of columns to export.

• You can export columns in an order different from the table schema.

• You can configure constants by providing the configuration in JSON format. The following is an example: ["id","1", "'bazhen.csy'", "null", "to_char(a + 1)", "2.3" , "true"].

  • id is a regular column name.

  • 1 is an integer constant.

  • 'bazhen.csy' is a string constant.

  • null is a null pointer.

  • to_char(a + 1) is a function expression.

  • 2.3 is a floating-point number.

  • true is a Boolean value.

• The column parameter must specify the columns that you want to sync. It cannot be empty.

Yes

None

splitFactor

The split factor. This parameter specifies the number of chunks into which the data is split for synchronization. If you configure multiple concurrent threads, the data is split into a number of chunks equal to Concurrency × splitFactor. For example, if Concurrency is set to 5 and splitFactor is set to 5, the data is split into 25 chunks (5 × 5) and processed by 5 concurrent threads.

splitPk

When Snowflake Reader extracts data, you can specify the splitPk parameter to shard data based on the specified primary key column. This allows the data synchronization system to run concurrent tasks to improve efficiency.

• Set splitPk to the primary key of the table. The values in a primary key column are typically distributed evenly. This prevents data hot spots in the shards.

• Currently, the splitPk parameter supports only integer columns for data sharding. If you specify a column of another data type, such as string, floating-point number, or date, Snowflake Reader reports an error.

where

The filter condition. Snowflake Reader combines the specified column, table, and where parameters into an SQL statement to extract data. For example, for testing purposes, you can set the where condition to limit 10.

For incremental synchronization, you can set the where condition to gmt_create>${bizdate}.

• The where condition allows for efficient incremental data synchronization.

• If you do not configure this parameter or leave it empty, all data is synced from the table.

No

None

fetchSize

This parameter specifies the number of records to retrieve in each batch from the database server. A larger value reduces the number of network interactions between Data Integration and the server, which improves data extraction performance.

No

1024

Parameter

Description

Required

Default value

datasource

The name of the Snowflake data source.

Yes

None

table

The name of the table to which you want to sync data.

Yes

None

schema

The schema where the Snowflake table that you want to write to resides.

Yes

None

column

The columns in the destination table to which you want to write data. Separate the column names with commas (,). Example: "column":["id","name","age"]. To write data to all columns in order, use an asterisk (*). Example: "column":["*"].

Yes

None

batchSize

The number of records to write in a single batch. A larger value can significantly reduce network interactions between Data Integration and Snowflake and improve overall throughput. However, a value that is too large may cause an OOM error in the Data Integration process.

No

1024