Hologres Writer writes data to Hologres and helps you analyze the data in real time.

Notice Hologres Writer supports only exclusive resource groups for Data Integration, but not the shared resource group or custom resource groups for Data Integration. For more information, see Create and use an exclusive resource group for Data Integration, Use a shared resource group, and Create a custom resource group for Data Integration.

Limits

Hologres Writer cannot write data to the external tables of Hologres.

How it works

Hologres Writer obtains data from a reader and writes data to Hologres based on the values of the writeMode and conflictMode parameters.
  • If you set the writeMode parameter to sdk, Hologres Writer writes data to Hologres by using the HoloHub API. This write mode can help implement optimal data synchronization performance.
  • If you set the writeMode parameter to insert, Hologres Writer writes data to Hologres by using Java Database Connectivity (JDBC) to execute the INSERT INTO statement that is provided by PostgreSQL. We recommend that you use this write mode.
You can use the conflictMode parameter to specify how to process conflicting data if a primary key conflict occurs.
  • If you set the conflictMode parameter to replace, new data overwrites existing data.
  • If you set the conflictMode parameter to ignore, existing data is retained, and new data is ignored.
In different write modes, different methods are used to process the conflicting data. If you set the writeMode parameter to sdk, you can configure the properties of the destination Hologres table to change the method that is used to process the conflicting data.
Notice The conflictMode parameter is suitable only for tables that have primary keys.

Parameters

Parameter Description Required Default value
endpoint The endpoint of the destination Hologres instance. Specify the value in the format of instance-id-region-endpoint.hologres.aliyuncs.com:Port number. You can view the endpoint of a Hologres instance on the configuration page of the instance in the Hologres console.
The endpoint of a Hologres instance varies based on the network type. Network types include the classic network, Internet, and virtual private cloud (VPC). Set this parameter based on the type of the network where your exclusive resource group for Data Integration and the Hologres instance reside. If an invalid endpoint is specified, the connection between the exclusive resource group for Data Integration and the Hologres instance may fail, or data synchronization performance may be poor. The endpoints for the three network types are in the following formats:
  • Public endpoint: instance-id-region-endpoint.hologres.aliyuncs.com:Port number
  • Classic network endpoint: instance-id-region-endpoint-internal.hologres.aliyuncs.com:Port number
  • VPC endpoint: instance-id-region-endpoint-vpc.hologres.aliyuncs.com:Port number

We recommend that you deploy the exclusive resource group for Data Integration and the Hologres instance in the same zone of the same region. This helps ensure a successful network connection and optimal data synchronization performance.

Yes No default value
accessId The AccessKey ID of the account that you use to connect to Hologres. Yes No default value
accessKey The AccessKey secret of the account that you use to connect to Hologres. Make sure that the account is authorized to write data to the destination table. Yes No default value
database The name of the destination database in the Hologres instance. Yes No default value
table The name of the destination Hologres table. You can specify the table name in the format of Schema name.Table name. Yes No default value
writeMode The write mode. Valid values: sdk and insert. For more information about the valid values, see How it works.
In the code editor, you can set the following parameters if you use the sdk mode:
  • maxCommitSize: the maximum size of data that Hologres Writer can write to Hologres at a time. Unit: bytes. Default value: 1048576. This parameter is optional.
  • maxRetryCount: the maximum number of retries allowed if a data write error occurs. Default value: 500. This parameter is optional.
  • retryInterval: the interval at which Hologres Writer performs retries. Unit: milliseconds. Default value: 1000. This parameter is optional.
Yes No default value
conflictMode The mode in which the conflicting data is processed. Valid values: replace and ignore. For more information about the valid values, see How it works. Yes No default value
column The names of the columns to which you want to write data. The names of the primary key columns in the destination table must be included. If you want to write data to all the columns in the destination table, set this parameter to ["*"]. Yes No default value
partition The destination partition. Specify the partition in the format of partition key column=value. This parameter is valid only for partitioned tables.
Note
  • Hologres supports only list partitioning, and you can specify only one column as the partition key column. The data type of the partition key column must be INT4 or TEXT.
  • The value of this parameter must match the partition configuration in the data definition language (DDL) statements that are used to create the Hologres table.
No Left empty, which indicates that the destination table is a non-partitioned table

Configure Hologres Writer by using the codeless UI

  1. Configure data sources.
    Configure Source and Target for the synchronization node. Configure data sources
    Parameter Description
    Connection The name of the data source to which you want to write data.
    Table The name of the table to which you want to write data. This parameter is equivalent to the table parameter that is described in the preceding section.
    Write Mode The write mode. This parameter is equivalent to the writeMode parameter that is described in the preceding section.
    Write conflict strategy The write conflict strategy. This parameter is equivalent to the conflictMode parameter that is described in the preceding section.
  2. Configure field mapping. This operation is equivalent to setting the column parameter that is described in the preceding section. Fields in the source on the left have a one-to-one mapping with fields in the destination on the right. Field mappings
    Operation Description
    Map Fields with the Same Name Click Map Fields with the Same Name to establish mappings between fields with the same name. The data types of the fields must match.
    Map Fields in the Same Line Click Map Fields in the Same Line to establish mappings between fields in the same row. The data types of the fields must match.
    Delete All Mappings Click Delete All Mappings to remove mappings that are established.
    Auto Layout Click Auto Layout. Then, the system automatically sorts the fields based on specific rules.

Configure Hologres Writer by using the code editor

For more information about how to configure a synchronization node by using the code editor, see Create a sync node by using the code editor. The following sample code provides examples on how to configure a synchronization node to write data to a non-partitioned table and how to configure a synchronization node to write data to a partitioned table.
  • Write data to a non-partitioned table
    • In the following code, a synchronization node is configured to write the data obtained from a reader to a non-partitioned Hologres table in sdk mode:
      {
          "type":"job",
          "version":"2.0",// The version number. 
          "steps":[
              {
                  "stepType":"stream",
                  "parameter":{},
                  "name":"Reader",
                  "category":"reader"
              },
              {
                  "stepType":"holo",
                  "parameter":{
                    "endpoint": "instance-id-region-endpoint.hologres.aliyuncs.com:port",
                      "accessId": "<yourAccessKeyId>", // The AccessKey ID of the account that you use to connect to Hologres. 
                      "accessKey": "<yourAccessKeySecret>", // The AccessKey secret of the account that you use to connect to Hologres. 
                      "database": "postgres",
                      "table": "<yourTableName>",
                      "writeMode": "sdk",
                      "conflictMode": "replace",
                      "column" : [
                          "tag",
                          "id",
                          "title"
                      ],
                      "maxCommitSize": 1048576,
                      "maxRetryCount": 500
                  },
                  "name":"Writer",
                  "category":"writer"
              }
          ],
          "setting":{
              "errorLimit":{
                  "record":"0"// The maximum number of dirty records allowed. 
              },
              "speed":{
                      "throttle": true,// Specifies whether to enable bandwidth throttling. The value false indicates that bandwidth throttling is disabled, and the value true indicates that bandwidth throttling is enabled. The mbps parameter takes effect only when the throttle parameter is set to true. 
                      "concurrent":1, // The maximum number of parallel threads. 
                      "mbps":"12"// The maximum transmission rate.
              }
          },
          "order":{
              "hops":[
                  {
                      "from":"Reader",
                      "to":"Writer"
                  }
              ]
          }
      }
    • The following sample DDL statements are used to create a non-partitioned Hologres table:
      begin;
      drop table if exists test_holowriter_sdk_replace;
      create table test_holowriter_sdk_replace(
        tag text not null, 
        id int not null, 
        body text not null
        primary key (tag, id));
        call set_table_property('test_holowriter_sdk_replace', 'orientation', 'column');
        call set_table_property('test_holowriter_sdk_replace', 'shard_count', '3');
      commit;
  • Write data to a partition in a partitioned table
    • In the following code, a synchronization node is configured to write the data obtained from a reader to a partition in a partitioned Hologres table in sdk mode:
      Note Exercise caution when you set the partition parameter.
      {
          "type":"job",
          "version":"2.0",// The version number. 
          "steps":[
              {
                  "stepType":"stream",
                  "parameter":{},
                  "name":"Reader",
                  "category":"reader"
              },
              {
                  "stepType":"holo",
                  "parameter":{
                    "endpoint": "instance-id-region-endpoint.hologres.aliyuncs.com:port",
                      "accessId": "<yourAccessKeyId>", // The AccessKey ID of the account that you use to connect to Hologres. 
                      "accessKey": "<yourAccessKeySecret>", // The AccessKey secret of the account that you use to connect to Hologres. 
                      "database": "postgres",
                      "table": "<yourTableName>",
                      "writeMode": "sdk",
                      "conflictMode": "ignore",
                      "column" : [
                          "*"
                      ],
                      "partition": "tag=foo",
                      "maxCommitSize": 1048576,
                      "maxRetryCount": 500
                  },
                  "name":"Writer",
                  "category":"writer"
              }
          ],
          "setting":{
              "errorLimit":{
                  "record":"0"// The maximum number of dirty records allowed. 
              },
              "speed":{
                  "throttle": true,// Specifies whether to enable bandwidth throttling. The value false indicates that bandwidth throttling is disabled, and the value true indicates that bandwidth throttling is enabled. The mbps parameter takes effect only when the throttle parameter is set to true. 
                  "concurrent":1, // The maximum number of parallel threads. 
                  "mbps":"12"// The maximum transmission rate.
              }
          },
          "order":{
              "hops":[
                  {
                      "from":"Reader",
                      "to":"Writer"
                  }
              ]
          }
      }
    • The following sample DDL statements are used to create a partitioned Hologres table:
      begin;
      drop table if exists test_holowriter_part_table_sdk_ignore;
      create table test_holowriter_part_table_sdk_ignore(
        tag text not null, 
        id int not null, 
        title text not null, 
        body text, 
        primary key (tag, id))
        partition by list( tag );
        call set_table_property('test_holowriter_part_table_sdk_ignore', 'orientation', 'column');
        call set_table_property('test_holowriter_part_table_sdk_ignore', 'shard_count', '3');
      commit;