Synchronize data to Hologres

Preparations

1. Create a project

Log on to the DataHub console. In the left-side navigation pane, click Project Manager.
On the Project List page, click Create Project in the upper-right corner.
In the Create Project panel, set the parameters as required and click Create.

2. Create a topic

On the Project List page, find the created project and click the project name.
On the details page of the project, click Create Topic in the upper-right corner. In the Create Topic panel, set the parameters as required.
Note: You can synchronize only data of the TUPLE type from DataHub to Hologres.

The following table lists the fields of a sample topic.

Index	Field	Type	NULL value allowed
0	l_orderkey	BIGINT	False
1	l_partkey	BIGINT	False
2	l_suppkey	BIGINT	False
3	l_linenumber	BIGINT	False
4	l_quantity	DECIMAL	True
5	l_extendedprice	DECIMAL	True
6	l_discount	DECIMAL	True
7	l_tax	DECIMAL	True
8	l_returnflag	STRING	True
9	l_linestatus	STRING	True
10	l_shipdate	TIMESTAMP	True
11	l_commitdate	TIMESTAMP	True
12	l_receiptdate	TIMESTAMP	True
13	l_shipinstruct	STRING	True
14	l_shipmode	STRING	True
15	l_comment	STRING	True

3. Write data

4. Create a table in Hologres for receiving data.

Create a table in Hologres for receiving data. You must create fields with the same names and data types as those of the DataHub topic. The following sample statements provide an example on how to create a table:

   BEGIN;
   CREATE TABLE lineitem ( 
   L_ORDERKEY BIGINT NOT NULL,
   L_PARTKEY BIGINT NOT NULL,
   L_SUPPKEY BIGINT NOT NULL,
   L_LINENUMBER BIGINT NOT NULL,
   L_QUANTITY DECIMAL(20,10),
   L_EXTENDEDPRICE DECIMAL(20,10),
   L_DISCOUNT DECIMAL(20,10),
   L_TAX DECIMAL(20,10),
   L_RETURNFLAG TEXT,
   L_LINESTATUS TEXT,
   L_SHIPDATE TIMESTAMPTZ,
   L_COMMITDATE TIMESTAMPTZ,
   L_RECEIPTDATE TIMESTAMPTZ,
   L_SHIPINSTRUCT TEXT,
   L_SHIPMODE TEXT,
   L_COMMENT TEXT
   );

   CALL set_table_property('lineitem', 'orientation', 'column');
   CALL set_table_property('lineitem', 'shard_count', '8');

   COMMIT;

5. Create a DataConnector

On the Topic List tab of the project details page, find the created topic and click the topic name.
On the details page of the topic, click Connector in the upper-right corner.
In the Create Connector panel, click Hologres. Then, set the parameters as required and click Create.
You can synchronize data to Hologres partition tables. After you create partition tables in Hologres, data is automatically synchronized to the partition tables based on partition fields.

Parameter	Description	Remarks
Instance	The ID of the Hologres instance.	You can view the instance ID in the Hologres console.
DataBase	The name of the database in the Hologres instance.	N/A
Table	The name of the Hologres table for receiving data.	N/A
Import Fields	The fields to be synchronized to the Hologres table.	You can synchronize all or partial fields of the DataHub topic based on your business requirements.
Auth Mode	The mode in which the access to the Hologres table is authenticated. Default value: AK.	N/A
AccessId	The AccessKey ID of the current Alibaba Cloud account used to access the Hologres instance.	You can obtain the AccessKey ID from the Security Management page.
AccessKey	The AccessKey secret of the current Alibaba Cloud account used to access the Hologres instance.	You can obtain the AccessKey secret from the Security Management page.

6.Wait until the data is synchronized to Hologres.

7.Query the data in Hologres

Connect the Hologres instance to a development tool and use the tool to check whether the data is synchronized to the Hologres instance in real time. For more information about development tools, see Overview. For example, you can execute the following statement to query the synchronized data:

SELECT COUNT(*) FROM lineitem;

Data type mappings

The following table lists the mappings between data types in DataHub and Hologres.

DataHub	Hologres
BIGINT	BIGINT
STRING	TEXT
BOOLEAN	BOOLEAN
DOUBLE	DOUBLE PRECISION
TIMESTAMP	TIMESTAMPTZ
DECIMAL	DECIMAL