This topic describes the data types and parameters supported by KingbaseES Reader and how to configure it by using the codeless user interface (UI) or code editor. Before you create a data integration node, you can refer to this topic to understand the data types and parameters that you must configure for KingbaseES Reader to extract data from data sources.
Background information
KingbaseES Reader connects to a remote KingbaseES database by using Java Database Connectivity (JDBC), generates a SELECT statement based on your configurations, and then sends the statement to the KingbaseES database. The KingbaseES database executes the statement and returns the result. After the result is returned, KingbaseES Reader assembles the returned data to abstract datasets of custom data types that are supported by the data integration service. Then, KingbaseES Reader passes the datasets to a writer.
Data types
Data type | The data type of SAP HANA |
---|---|
Integer | INT, TINYINT, SMALLINT, MEDIUMINT, and BIGINT |
Floating point | FLOAT, DOUBLE, and DECIMAL |
String | VARCHAR, CHAR, TINYTEXT, TEXT, MEDIUMTEXT, and LONGTEXT |
Date and time | DATE, DATETIME, TIMESTAMP, TIME, and YEAR |
Boolean | BIT and BOOL |
Binary | TINYBLOB, MEDIUMBLOB, BLOB, LONGBLOB, and VARBINARY |
- Data types other than those specified in the preceding table are not supported.
- KingbaseES Reader considers TINYINT(1) as an integer type.
Parameters
Parameter | Description |
---|---|
username | The username that you can use to connect to SAP HANA. |
password | The password that you can use to connect to SAP HANA. |
column | The columns to be synchronized. Set this parameter to an asterisk (*) if all the columns in the source table need to be synchronized. |
table | The name of the table to be synchronized. |
jdbcUrl | The Java Database Connectivity (JDBC) URL of SAP HANA. Example: jdbc:sap://127.0.0.1:30215? currentschema=TEST. |
splitPk | The field that is used for data sharding when SAP HANA Reader extracts data. If you
specify the splitPk parameter, the table is sharded based on the shard key that is
specified by this parameter. Data Integration then runs concurrent threads to synchronize
data.
The splitPk parameter supports data sharding only for integers. If no integer fields exist, do not specify this parameter. |
Configure KingbaseES Reader by using the codeless UI
- Configure a data source.
Configure a data source for Source.
Parameter Description Data source The datasource parameter in the preceding parameter description. Select the name of the data source that you have configured. Table The table parameter in the preceding parameter description. Filter The condition for filtering the data that you want to synchronize. SAP HANA Reader cannot filter data based on the limit keyword. The SQL syntax is determined by the selected data source. Shard Key The shard key. You can specify a column in the source table as the shard key. We recommend that you use the primary key or an indexed column as the shard key. Only integer fields are supported. If data sharding is performed based on the configured shard key, data can be read concurrently to improve data synchronization efficiency.Note The Shard Key parameter is displayed only when you configure the source connection for a sync node.
Configure KingbaseES Reader by using the code editor
- Configure a synchronization node to read data from a database or table that is not
sharded
{ "type":"job", "version":"2.0",// The version number. "steps":[ { "stepType":"kingbasees",// The reader type. "parameter":{ "column":[// The columns to be synchronized. "id" ], "connection":[ { "querySql":["select a,b from join1 c join join2 d on c.id = d.id;"], // Specify the querySql parameter in the connection parameter as a string. "datasource":"",// The data source. "table":[// The name of the table to be synchronized. The table name must be enclosed in brackets []. "xxx" ] } ], "where":"",// The WHERE clause. "splitPk":"",// The shard key. "encoding":"UTF-8"// The encoding format. }, "name":"Reader", "category":"reader" }, { "stepType":"stream", "parameter":{}, "name":"Writer", "category":"writer" } ], "setting":{ "errorLimit":{ "record":"0"// The maximum number of dirty data records allowed. }, "speed":{ "throttle":false,// Specifies whether to enable throttling. The value false indicates that throttling is disabled, and the value true indicates that throttling is enabled. The throttle parameter must be set to true before the concurrent parameter takes effect. "concurrent":1,// The maximum number of concurrent jobs. } }, "order":{ "hops":[ { "from":"Reader", "to":"Writer" } ] } }
- Configure a synchronization node to read data from a database or table that is sharded
Note For sharded databases and tables, KingbaseES Reader can read data from multiple tables with the same schema.
{ "type": "job", "version": "1.0", "configuration": { "reader": { "plugin": "kingbasees", "parameter": { "connection": [ { "table": [ "tbl1", "tbl2", "tbl3" ], "datasource": "datasourceName1" }, { "table": [ "tbl4", "tbl5", "tbl6" ], "datasource": "datasourceName2" } ], "singleOrMulti": "multi", "splitPk": "db_id", "column": [ "id", "name", "age" ], "where": "1 < id and id < 100" } }, "writer": { } } }