All Products
Search
Document Center

DataWorks:Configure Hologres Writer

Last Updated:Aug 01, 2023

You can build a real-time data warehouse by using the real-time write capability of Hologres.

Prerequisites

A reader or conversion node is configured. For more information, see Data source types that support real-time synchronization.

Background information

  • The Hologres data sources of the following versions are supported: V0.7, V0.8, V0.9, V0.10, and V.1.1.

  • UUID fields cannot be synchronized.

Procedure

  1. Go to the DataStudio page.

    1. Log on to the DataWorks console.

    2. In the left-side navigation pane, click Workspaces.

    3. In the top navigation bar, select the region in which the workspace that you want to manage resides. On the Workspaces page, find the workspace and click Shortcuts > Data Development in the Actions column.

  2. In the Scheduled Workflow pane, move the pointer over the Create a table icon and choose Create Node > Data Integration > Real-time synchronization.

    Alternatively, right-click the required workflow, and then choose Create Node > Data Integration > Real-time synchronizationReal-time synchronization.

  3. In the Create Node dialog box, set the Sync Method parameter to End-to-end ETL and configure the Name and Path parameters.

    Important

    The node name cannot exceed 128 characters in length and can contain letters, digits, underscores (_), and periods (.).

  4. Click Confirm.

  5. On the configuration tab of the real-time synchronization node, drag Hologres in the Output section to the canvas on the right. Then, draw a line to connect it to the configured reader or conversion node.

  6. Click the Hologres node. In the panel that appears, configure the parameters.

    Parameter

    Description

    Data source

    The Hologres data source that you configured. You can select only a Hologres data source.

    If no data source is available, click New data source on the right to add a data source on the Data Source page. For more information, see Add a Hologres data source.

    Table

    The name of the Hologres table to which you want to write data.

    You can click Create Table on the right to create a table, or click Data preview to preview the selected table.

    Dynamic Time Partition

    If the Hologres table is a partitioned table, you must specify a dynamic time-based partition.

    The dynamic time-based partition parses the value of a source field in the yyyymmdddhhmmss format. After the value is parsed, you can use the dynamic partition whose name is a string of variables in the destination table. The destination partition varies based on the value of the source field.

    For example, the value of the source field is 20200816, and the name of the destination partition is in the {yyyy}-{mm}-{dd} format. In this case, the value is written to the 2020-08-16 partition.

    Job Type

    The type of the data write operation. Valid values: Replay (replay operation log to restore data) and Insert (direct archive save).

    • Replay (replay operation log to restore data): Hologres Writer performs the same operation on the destination as that performed on the source. For example, if the INSERT statement is executed to add a data record to the source, Hologres Writer executes the INSERT statement to add the same data record to the destination. If the UPDATE or DELETE statement is executed in the source, Hologres Writer executes the UPDATE or DELETE statement in the destination.

    • Insert (direct archive save): Hologres Writer uses the destination as streaming data storage. Data is synchronized from the source to the destination by using the INSERT statement.

    Policy for Write Conflict

    The solution to data write conflicts. Valid values: Cover (Overwrite) and Ignore (Ignore).

    • Cover (Overwrite): Hologres Writer uses the new data synchronized from the source to overwrite the existing data in the destination.

    • Ignore (Ignore): Hologres Writer ignores the new data synchronized from the source and retains the existing data in the destination.

    Field Mapping

    The field mappings between the source and destination. Click Field Mapping to configure field mappings. The synchronization node synchronizes data based on the field mappings.

  7. Click the Save icon in the top toolbar to save the configurations.