You can build a real-time data warehouse by using the real-time write capability of Hologres.
Prerequisites
A reader or conversion node is configured. For more information, see Data source types that support real-time synchronization.
Background information
The Hologres data sources of the following versions are supported: V0.7, V0.8, V0.9, V0.10, and V.1.1.
UUID fields cannot be synchronized.
Procedure
Go to the DataStudio page.
Log on to the DataWorks console.
In the left-side navigation pane, click Workspaces.
In the top navigation bar, select the region in which the workspace that you want to manage resides. On the Workspaces page, find the workspace and click in the Actions column.
In the Scheduled Workflow pane, move the pointer over the icon and choose .
Alternatively, right-click the required workflow, and then choose
.In the Create Node dialog box, set the Sync Method parameter to End-to-end ETL and configure the Name and Path parameters.
ImportantThe node name cannot exceed 128 characters in length and can contain letters, digits, underscores (_), and periods (.).
Click Confirm.
On the configuration tab of the real-time synchronization node, drag Hologres in the Output section to the canvas on the right. Then, draw a line to connect it to the configured reader or conversion node.
Click the Hologres node. In the panel that appears, configure the parameters.
Parameter
Description
Data source
The Hologres data source that you configured. You can select only a Hologres data source.
If no data source is available, click New data source on the right to add a data source on the Data Source page. For more information, see Add a Hologres data source.
Table
The name of the Hologres table to which you want to write data.
You can click Create Table on the right to create a table, or click Data preview to preview the selected table.
Dynamic Time Partition
If the Hologres table is a partitioned table, you must specify a dynamic time-based partition.
The dynamic time-based partition parses the value of a source field in the yyyymmdddhhmmss format. After the value is parsed, you can use the dynamic partition whose name is a string of variables in the destination table. The destination partition varies based on the value of the source field.
For example, the value of the source field is 20200816, and the name of the destination partition is in the {yyyy}-{mm}-{dd} format. In this case, the value is written to the 2020-08-16 partition.
Job Type
The type of the data write operation. Valid values: Replay (replay operation log to restore data) and Insert (direct archive save).
Replay (replay operation log to restore data): Hologres Writer performs the same operation on the destination as that performed on the source. For example, if the
INSERT
statement is executed to add a data record to the source, Hologres Writer executes theINSERT
statement to add the same data record to the destination. If theUPDATE
orDELETE
statement is executed in the source, Hologres Writer executes theUPDATE
orDELETE
statement in the destination.Insert (direct archive save): Hologres Writer uses the destination as streaming data storage. Data is synchronized from the source to the destination by using the
INSERT
statement.
Policy for Write Conflict
The solution to data write conflicts. Valid values: Cover (Overwrite) and Ignore (Ignore).
Cover (Overwrite): Hologres Writer uses the new data synchronized from the source to overwrite the existing data in the destination.
Ignore (Ignore): Hologres Writer ignores the new data synchronized from the source and retains the existing data in the destination.
Field Mapping
The field mappings between the source and destination. Click Field Mapping to configure field mappings. The synchronization node synchronizes data based on the field mappings.
Click the icon in the top toolbar to save the configurations.