This topic describes the data types and parameters that are supported by KingbaseES Reader and how to configure KingbaseES Reader by using the codeless user interface (UI) and code editor. Before you create a Data Integration node, you can refer to this topic to familiarize yourself with the data types and parameters that you must configure for KingbaseES Reader to read data from data sources.

Background information

KingbaseES Reader connects to a remote KingbaseES database by using Java Database Connectivity (JDBC), generates an SQL statement based on your configurations, and then sends the statement to the database. The system executes the statement on the database and returns data. Then, KingbaseES Reader assembles the returned data into abstract datasets of the data types supported by Data Integration and sends the datasets to a writer.

Data types

The following table lists the data types that are supported by KingbaseES Reader.
CategorySAP KingbaseES data type
IntegerINT, TINYINT, SMALLINT, MEDIUMINT, and BIGINT
Floating pointFLOAT, DOUBLE, and DECIMAL
StringVARCHAR, CHAR, TINYTEXT, TEXT, MEDIUMTEXT, and LONGTEXT
Date and timeDATE, DATETIME, TIMESTAMP, TIME, and YEAR
BooleanBIT and BOOLEAN
BinaryTINYBLOB, MEDIUMBLOB, BLOB, LONGBLOB, and VARBINARY
Important
  • Data types that are not listed in the preceding table are not supported.
  • KingbaseES Reader processes TINYINT(1) as an integer data type.

Parameters

ParameterDescription
usernameThe username that you can use to connect to SAP HANA.
passwordThe password that you can use to connect to SAP HANA.
columnThe columns to be synchronized. Set this parameter to an asterisk (*) if all the columns in the source table need to be synchronized.
tableThe name of the table to be synchronized.
jdbcUrlThe Java Database Connectivity (JDBC) URL of SAP HANA. Example: jdbc:sap://127.0.0.1:30215? currentschema=TEST.
splitPkThe field that is used for data sharding when SAP HANA Reader extracts data. If you specify the splitPk parameter, the table is sharded based on the shard key that is specified by this parameter. Data Integration then runs concurrent threads to synchronize data.

The splitPk parameter supports data sharding only for integers. If no integer fields exist, do not specify this parameter.

Configure KingbaseES Reader by using the codeless UI

  1. Configure data sources.
    Configure Source and Target for the synchronization node. Configure a KingbaseES data source
    ParameterDescription
    ConnectionThe name of the data source from which you want to read data. This parameter is equivalent to the datasource parameter that is specified in the preceding section.
    TableThe name of the table from which you want to read data. This parameter is equivalent to the table parameter that is specified in the preceding section.
    FilterThe condition that is used to filter the data you want to read. Filtering based on the LIMIT keyword is not supported. The SQL syntax is determined by the selected data source.
    Split pkThe shard key. You can use a column in the source table as the shard key. We recommend that you use the primary key column or an indexed column. Only integer columns are supported.
    If you configure this parameter, data sharding is performed based on the value of this parameter, and parallel threads can be used to read data. This improves data synchronization efficiency.
    Note The Split pk parameter is displayed only after you select the data source for the synchronization node.
  2. Configure field mappings.
    Fields in the source on the left have a one-to-one mapping with fields in the destination on the right. You can click Add to add a field. To remove an added field, move the pointer over the field and click the Remove icon. Field mappings
    OperationDescription
    Map Fields with the Same NameClick Map Fields with the Same Name to establish mappings between fields with the same name. The data types of the fields must match.
    Map Fields in the Same LineClick Map Fields in the Same Line to establish mappings between fields in the same row. The data types of the fields must match.
    Delete All MappingsClick Delete All Mappings to remove the mappings that are established.
    Auto LayoutClick Auto Layout. Then, the system automatically sorts the fields based on specific rules.
    Change FieldsClick the Change Fields icon. In the Change Fields dialog box, you can manually edit the fields in the source table. Each field occupies a row. The first and the last blank rows are included, whereas other blank rows are ignored.
    AddClick Add to add a field. Take note of the following rules when you add a field:
    • You can enter constants. Each constant must be enclosed in single quotation marks ('), such as 'abc' and '123'.
    • You can use scheduling parameters, such as ${bizdate}.
    • You can enter functions that are supported by relational databases, such as now() and count(1).
    • If the field that you entered cannot be parsed, the value of Type for the field is Unidentified.
  3. Configure channel control policies. Channel control
    ParameterDescription
    Expected Maximum ConcurrencyThe maximum number of concurrent threads that the synchronization node can use to read data from the source or write data to the destination. You can configure the concurrency for the synchronization node on the codeless UI.
    Bandwidth ThrottlingSpecifies whether to enable throttling. You can enable throttling and specify a maximum transmission rate to prevent heavy read workloads on the source. We recommend that you enable throttling and set the maximum transmission rate to an appropriate value based on the configurations of the source.
    Dirty Data Records AllowedThe maximum number of dirty data records allowed.
    Distributed ExecutionThis parameter is not supported for data synchronization nodes that use RestAPI Reader.

Configure KingbaseES Reader by using the code editor

The following sample code provides examples on how to configure a synchronization node to read data from a table that is not sharded and how to configure a synchronization node to read data from a sharded table.
  • Configure a synchronization node to read data from a table that is not sharded
    {
    "type":"job",
    "version":"2.0",// The version number. 
    "steps":[
        {
            "stepType":"kingbasees",// The reader type. 
            "parameter":{
                "column":[// The names of the columns from which you want to read data. 
                    "id"
                ],
                "connection":[
                    {   "querySql":["select a,b from join1 c join join2 d on c.id = d.id;"], // The SQL statement that is used to read data from the source table. 
                        "datasource":"",// The name of the data source. 
                        "table":[// The name of the source table. The table name must be enclosed in brackets []. 
                            "xxx"
                        ]
                    }
                ],
                "where":"",// The WHERE clause. 
                "splitPk":"",// The shard key. 
                "encoding":"UTF-8"// The encoding format. 
            },
            "name":"Reader",
            "category":"reader"
        },
        {
            "stepType":"stream",
            "parameter":{},
            "name":"Writer",
            "category":"writer"
        }
    ],
    "setting":{
        "errorLimit":{
            "record":"0"// The maximum number of dirty data records allowed. 
        },
        "speed":{
            "throttle":true,// Specifies whether to enable bandwidth throttling. The value false indicates that bandwidth throttling is disabled, and the value true indicates that bandwidth throttling is enabled. The mbps parameter takes effect only when the throttle parameter is set to true. 
            "concurrent":1, // The maximum number of parallel threads. 
            "mbps":"12"// The maximum transmission rate.
        }
    },
    "order":{
        "hops":[
            {
                "from":"Reader",
                "to":"Writer"
            }
        ]
    }
}
  • Configure a synchronization node to read data from a sharded table
    Note When you configure a synchronization node to read data from a sharded KingbaseES table, you can select multiple partitions that have the same schema. 
    {
    "type": "job",
    "version": "1.0",
    "configuration": {
        "reader": {
            "plugin": "kingbasees",
            "parameter": {
                "connection": [
                    {
                        "table": [
                            "tbl1",
                            "tbl2",
                            "tbl3"
                        ],
                        "datasource": "datasourceName1"
                    },
                    {
                        "table": [
                            "tbl4",
                            "tbl5",
                            "tbl6"
                        ],
                        "datasource": "datasourceName2"
                    }
                ],
                "singleOrMulti": "multi",
                "splitPk": "db_id",
                "column": [
                    "id", "name", "age"
                ],
                "where": "1 < id and id < 100"
            }
        },
        "writer": {            
        }
    }
}