This topic describes the limits and parameters of Gbase 8a Writer and how to configure it by using the code editor.

Gbase 8a is a new type of column-oriented analytical database. Gbase 8a Writer allows you to write data to Gbase 8a databases.
Notice Gbase 8a Writer supports only exclusive resource groups for Data Integration, but not the default resource group or custom resource groups. For more information, see Use exclusive resource groups for data integration and Add a custom resource group.

Specifically, Gbase 8a Writer connects to a remote Gbase 8a database by using Java Database Connectivity (JDBC), and executes an INSERT INTO statement to write data to the Gbase 8a database. Internally, data is written to the database in batches.

Gbase 8a Writer is designed for extract, transform, load (ETL) developers to import data from data warehouses to Gbase 8a databases. Users such as database administrators can use Gbase 8a Writer as a data migration tool.

Gbase 8a Writer obtains data from a Data Integration reader, and generates an INSERT INTO statement based on your configurations.

Limits

  • INSERT INTO: If a primary key conflict or unique index conflict occurs, data cannot be written to the conflicting rows.
  • Gbase 8a Writer can write data to tables only in the primary Gbase 8a database.
    Note A sync node that uses Gbase 8a Writer must have at least the permission to execute the INSERT INTO statement. Whether other permissions are required depends on the SQL statements that you specify in the preSql and postSql parameters when you configure the node.
  • Gbase 8a Writer does not support the writeMode parameter.
  • Gbase 8a Writer connects to a Gbase 8a database by using the MySQL database driver. You must confirm the compatibility between the driver version and your Gbase 8a database. Gbase 8a Writer uses the following version of the MySQL database driver:
    <dependency>
        <groupId>mysql</groupId>
        <artifactId>mysql-connector-java</artifactId>
        <version>5.1.22</version>
    </dependency>

Parameters

Parameter Description Required Default value
jdbcUrl The JDBC URL of the Gbase 8a database. The jdbcUrl parameter must be included in the connection parameter.
  • You can configure only one JDBC URL for a database. Gbase 8a Writer cannot write data to a database with multiple primary databases.
  • The format must be in accordance with Gbase 8a official specifications. You can also specify the information of the attachment facility. Example: jdbc:mysql://127.0.0.1:3306/database.
Yes N/A
username The username that you can use to connect to the database. Yes N/A
password The password that you can use to connect to the database. Yes N/A
table The name of the destination table, which is described in a JSON array.
Note The table parameter must be included in the connection parameter.
Yes N/A
column The columns in the destination table to which data is written. Separate the columns with a comma (,), for example, "column":["id","name","age"].
Note The column parameter cannot be left empty.
Yes N/A
preSql The SQL statement to execute before the sync node is run. Use @table to specify the name of the destination table in the SQL statement. When you execute this SQL statement, DataWorks replaces @table with the name of the destination table. No N/A
postSql The SQL statement to execute after the sync node is run. No N/A
batchSize The number of data records to write at a time. Set this parameter to a proper value based on your actual needs. This can greatly reduce the interactions between Data Integration and the Gbase 8a database on the network, and increase the throughput. However, an excessively large value may lead to the out of memory (OOM) error during the data synchronization process. No 1024

Configure Gbase 8a Writer by using the codeless UI

The codeless user interface (UI) is not supported for Gbase 8a Writer.

Configure Gbase 8a Writer by using the code editor

You can configure Gbase 8a Writer by using the code editor. For more information, see Create a sync node by using the code editor.

The following example shows how to configure a sync node to write data to a Gbase 8a database:
{
    "type":"job",
    "version":"2.0",// The version number.
    "steps":[
        {
            "stepType":"stream",
            "parameter":{},
            "name":"Reader",
            "category":"reader"
        },
        {
            "stepType":"gbase8a",// The writer type.
            "parameter":{
                "datasource": "The connection name.",
                "username": "",
                "password": "",
                "column": [// The columns in the destination table to which data is written.
                    "id",
                    "name"
                ],
                "connection": [
                    {
                        "table":[// The name of the destination table.
                            "Gbase8a_table"
                        ],
                        "jdbcUrl": "jdbc:mysql://ip:port/database"
                    }
                ],
                "preSql": [ // The SQL statement to execute before the sync node is run.
                    "delete from @table where db_id = -1"
                ],
                "postSql": [// The SQL statement to execute after the sync node is run.
                    "update @table set db_modify_time = now() where db_id = 1"
                ]
            },
            "name":"Writer",
            "category":"writer"
        }
    ],
    "setting":{
        "errorLimit":{
            "record":"0"// The maximum number of dirty data records allowed.
        },
        "speed":{
            "throttle":false,// Specifies whether to enable bandwidth throttling. A value of false indicates that the bandwidth is not throttled. A value of true indicates that the bandwidth is throttled. The maximum transmission rate takes effect only if you set this parameter to true.
            "concurrent":1 // The maximum number of concurrent threads.
        }
    },
    "order":{
        "hops":[
            {
                "from":"Reader",
                "to":"Writer"
            }
        ]
    }
}