All Products
Search
Document Center

Synchronize data to Hologres

Last Updated: Aug 25, 2021

Synchronize data to Hologres

Preparations

1. Create a project

  1. Log on to the DataHub console. In the left-side navigation pane, click Project Manager.

  2. On the Project List page, click Create Project in the upper-right corner.

  3. In the Create Project panel, set the parameters as required and click Create.

2. Create a topic

  1. On the Project List page, find the created project and click the project name.

  2. On the details page of the project, click Create Topic in the upper-right corner. In the Create Topic panel, set the parameters as required.

  3. Note: You can synchronize only data of the TUPLE type from DataHub to Hologres.

1

The following table lists the fields of a sample topic.

Index

Field

Type

NULL value allowed

0

l_orderkey

BIGINT

False

1

l_partkey

BIGINT

False

2

l_suppkey

BIGINT

False

3

l_linenumber

BIGINT

False

4

l_quantity

DECIMAL

True

5

l_extendedprice

DECIMAL

True

6

l_discount

DECIMAL

True

7

l_tax

DECIMAL

True

8

l_returnflag

STRING

True

9

l_linestatus

STRING

True

10

l_shipdate

TIMESTAMP

True

11

l_commitdate

TIMESTAMP

True

12

l_receiptdate

TIMESTAMP

True

13

l_shipinstruct

STRING

True

14

l_shipmode

STRING

True

15

l_comment

STRING

True

3. Write data

4

4. Create a table in Hologres for receiving data.

Create a table in Hologres for receiving data. You must create fields with the same names and data types as those of the DataHub topic. The following sample statements provide an example on how to create a table:

   BEGIN;
   CREATE TABLE lineitem ( 
   L_ORDERKEY BIGINT NOT NULL,
   L_PARTKEY BIGINT NOT NULL,
   L_SUPPKEY BIGINT NOT NULL,
   L_LINENUMBER BIGINT NOT NULL,
   L_QUANTITY DECIMAL(20,10),
   L_EXTENDEDPRICE DECIMAL(20,10),
   L_DISCOUNT DECIMAL(20,10),
   L_TAX DECIMAL(20,10),
   L_RETURNFLAG TEXT,
   L_LINESTATUS TEXT,
   L_SHIPDATE TIMESTAMPTZ,
   L_COMMITDATE TIMESTAMPTZ,
   L_RECEIPTDATE TIMESTAMPTZ,
   L_SHIPINSTRUCT TEXT,
   L_SHIPMODE TEXT,
   L_COMMENT TEXT
   );

   CALL set_table_property('lineitem', 'orientation', 'column');
   CALL set_table_property('lineitem', 'shard_count', '8');

   COMMIT;

5. Create a DataConnector

  1. On the Topic List tab of the project details page, find the created topic and click the topic name.

  2. On the details page of the topic, click Connector in the upper-right corner.

  3. In the Create Connector panel, click Hologres. Then, set the parameters as required and click Create.

  4. You can synchronize data to Hologres partition tables. After you create partition tables in Hologres, data is automatically synchronized to the partition tables based on partition fields.

holo-1

Parameter

Description

Remarks

Instance

The ID of the Hologres instance.

You can view the instance ID in the Hologres console.

DataBase

The name of the database in the Hologres instance.

N/A

Table

The name of the Hologres table for receiving data.

N/A

Import Fields

The fields to be synchronized to the Hologres table.

You can synchronize all or partial fields of the DataHub topic based on your business requirements.

Auth Mode

The mode in which the access to the Hologres table is authenticated. Default value: AK.

N/A

AccessId

The AccessKey ID of the current Alibaba Cloud account used to access the Hologres instance.

You can obtain the AccessKey ID from the Security Management page.

AccessKey

The AccessKey secret of the current Alibaba Cloud account used to access the Hologres instance.

You can obtain the AccessKey secret from the Security Management page.

6.Wait until the data is synchronized to Hologres.

7.Query the data in Hologres

Connect the Hologres instance to a development tool and use the tool to check whether the data is synchronized to the Hologres instance in real time. For more information about development tools, see Overview. For example, you can execute the following statement to query the synchronized data:

SELECT COUNT(*) FROM lineitem;

Data type mappings

The following table lists the mappings between data types in DataHub and Hologres.

DataHub

Hologres

BIGINT

BIGINT

STRING

TEXT

BOOLEAN

BOOLEAN

DOUBLE

DOUBLE PRECISION

TIMESTAMP

TIMESTAMPTZ

DECIMAL

DECIMAL