DataWorks:PolarDB-X 2.0 data source

Parameter

Description

Required

Default value

datasource

The name of the data source. It must be the same as the name of the data source that you added on the data source management page.

Yes

No default value

table

The name of the table from which you want to read data. Each synchronization task can be used to synchronize data from only one table.

Yes

No default value

column

The names of the columns from which you want to read data. Specify the names in a JSON array. The default value is [*], which indicates all columns in the source table.

• You can select specific columns to read.

• The column order can be changed. This indicates that you can specify columns in an order different from the order specified by the schema of the source table.

• Constants are supported. The column names must be arranged in compliance with the SQL syntax supported by PolarDB-X 2.0, such as ["id","table","1","'mingya.wmy'","'null'","to_char(a+1)","2.3","true"].

  • id: a column name.

  • table: the name of a column that contains reserved keywords.

  • 1: an integer constant.

  • 'mingya.wmy': a string constant, which is enclosed in single quotation marks (').

  • null:

    • " " indicates an empty string.

    • null indicates a null value.

    • 'null' indicates the string null.

  • to_char(a+1): a function expression that is used to calculate the length of a string.

  • 2.3: a floating-point constant.

  • true: a Boolean value.

• The column parameter must explicitly specify all the columns from which you want to read data. This parameter cannot be left empty.

Yes

No default value

splitPk

The field that is used for data sharding when PolarDB-X 2.0 Reader reads data. If you configure this parameter, the source table is sharded based on the value of this parameter. Then, parallel threads are run to read data. This way, data can be synchronized more efficiently.

• We recommend that you set the splitPk parameter to the name of the primary key column of the table. Data can be evenly distributed to different shards based on the primary key column, instead of being intensively distributed only to specific shards.

• The splitPk parameter must be set to a field of an integer data type. If you set the splitPk parameter to a field of an unsupported data type, such as a string, floating point, or date data type, the setting of this parameter is ignored, and a single thread is used to read data.

• If the splitPk parameter is not contained in code for PolarDB-X 2.0 Reader or is left empty, a single thread is used to read data.

No

No default value

where

The WHERE clause. For example, you can set this parameter to gmt_create>$bizdate to read the data that is generated on the current day.

• You can use the WHERE clause to read incremental data. If the where parameter is not provided or is left empty, PolarDB-X 2.0 Reader reads all data.

• Do not set the where parameter to LIMIT 10. This value does not conform to the constraints of PolarDB-X 2.0 on the SQL WHERE clause.

No

No default value

checkSlave

Specifies whether to check the latency of data synchronization between a primary instance and a read-only secondary instance. If you add a PolarDB-X 2.0 read-only instance as a data source, you must check the latency of data synchronization between the read-only instance and a primary instance before you run a data synchronization task. This helps prevent data loss caused by the latency.

No

true

slaveDelayLimit

The maximum latency of data synchronization between a primary instance and a read-only instance, in seconds. If the latency exceeds the value of this parameter, the data synchronization task fails. This helps prevent data loss caused by the latency.

No

30

Parameter

Description

Required

Default value

datasource

The name of the data source. It must be the same as the name of the data source that you added on the data source management page.

Yes

No default value

table

The name of the table to which you want to write data.

Yes

No default value

writeMode

The write mode. Valid values:

• insert into: If a primary key conflict or unique index conflict occurs, data cannot be written to the conflicting rows, and the data that is not written to these rows is regarded as dirty data. If you configure a synchronization task by using the code editor, set writeMode to insert.

• replace into: If no primary key conflict or unique index conflict occurs, the data is processed in the same way as that when you set this parameter to insert into. If a conflict occurs, the original rows are deleted, and new rows are inserted. This indicates that all fields of the original rows are replaced. If you create a synchronization task by using the code editor, set writeMode to replace.

No

insert

column

The names of the columns to which you want to write data. Separate the names with commas (,), such as "column": ["id", "name", "age"]. If you want to write data to all the columns in the destination table, set this parameter to an asterisk (*), such as "column": ["*"].

Yes

No default value

preSql

The SQL statement that you want to execute before the synchronization task is run. You can execute only one SQL statement on the codeless UI and multiple SQL statements in the code editor. For example, you can set this parameter to the following SQL statement to delete outdated data before the synchronization task is run:

-- tablename indicates the table name.
truncate table tablename;

No

No default value

postSql

The SQL statement that you want to execute after the synchronization task is run. You can execute only one SQL statement on the codeless UI and multiple SQL statements in the code editor. For example, you can set this parameter to the following statements to add a timestamp after the synchronization task is run:

-- tablename indicates the table name, and colname indicates the column name.
alter table tablename 
add colname timestamp DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP;

No

No default value

batchSize

The number of data records to write at a time. Set this parameter to an appropriate value based on your business requirements. This greatly reduces the interactions between Data Integration and PolarDB-X 2.0 and increases throughput. If you set this parameter to an excessively large value, an out of memory (OOM) error may occur during data synchronization.

No

256