The HBase output widget enables data writing to the HBase data source. When synchronizing data from other sources to HBase, you must configure the target data source after setting up the source data. This topic guides you through configuring the HBase output widget.
Prerequisites
You must have purchased and activated the high availability feature of DataService Studio or the tag service module to configure active/standby links for the data source.
An HBase data source should be created. For specific operations, see Create HBase Data Source.
To configure the properties of the HBase output widget, the account must possess write-through permission for the data source. If permission is lacking, it is necessary to request access to the data source. For detailed instructions, see Request, renew, and return data source permissions.
Procedure
On the Dataphin home page, select Development > Data Integration from the top menu bar.
In the integration page's top menu bar, select Project (Dev-Prod mode requires selecting Environment).
In the navigation pane on the left, click Batch Pipeline. In the Batch Pipeline list, click the offline pipeline you want to develop to access its configuration page.
Click Component Library in the upper right corner to open the Component Library panel.
In the Component Library panel's left-side navigation pane, select Output. Find the HBase component in the output widget list and drag it to the canvas.
Drag the
icon from the target input, transform, or flow widget to connect it to the HBase output widget.On the HBase output component card, click the
icon to open the HBase Output Configuration dialog box.
Configure the parameters in the Hbase Output Configuration dialog box.
Parameter
Description
Basic settings
Step Name
This is the name of the HBase output widget. Dataphin automatically generates the step name, and you can also modify it according to the business scenario. The naming convention is as follows:
Can only contain Chinese characters, letters, underscores (_), and numbers.
Cannot exceed 64 characters.
Datasource
In the data source drop-down list, all HBase-type data sources are displayed, including data sources for which you have write-through permission and those for which you do not. Click the
icon to copy the current data source name.For data sources without write-through permission, you can click Request after the data source to request write-through permission for the data source. For specific operations, see Request, renew, and return data source permissions.
If you do not have an HBase-type data source, click Create Data Source to create a data source. For specific operations, see Create HBase data source.
Select Link
When the high availability of the tag service feature is enabled and the selected HBase data source active/standby link is Dual Active/standby Link, you can choose Active Link or Standby Link for integration, which only affects the production data source.
Table
Select the target table for data synchronization.
File Encoding
Select file encoding. File Encoding includes UTF-8 and GBK.
Rowkey
Click Add to configure multiple Rowkeys for the HBase data source. By default, the constant data types String, Int, Boolean, Long, Float, and Short are supported as Rowkeys for the HBase data source.
Version Number Source Of Value
Supports current time, fixed time, or specified time column.
Current time: Use the current time as the version number source of the value.
Specified time: Fixed time requires configuring the Select Time parameter to select the version number source time of the value.
Specified time column: Specified time column requires configuring the Select Time Column parameter to select the time column in the table that can be used as the version number source of the value.
Field mapping
Input Field
Displays the input fields based on the output of the upstream widget.
Output Field
Displays the output fields. Dataphin supports configuring output fields through Batch Add and Create New Output Field:
Batch Add: Click Batch Add to support batch configuration in JSON or TEXT format.
Batch configuration in JSON format, for example:
// Example: [{"name": "user_id","type": "String"}, {"name": "user_name","type": "String"}]Notename represents the name of the imported field, and type represents its field type. For example,
"name":"user_id","type":"String"indicates that the field named user_id is imported and its field type is set to String.Batch configuration in TEXT format, for example:
// Example: user_id,String user_name,StringThe row delimiter is used to separate the information of each field. The default is a line feed (\n), and it supports line feed (\n), semicolon (;), or period (.).
The column delimiter is used to separate the field name and field type, with the default being a comma (,).
Create new output field.
Click + Create New Output Field, and fill in the Column and select Type according to the page prompts.
Copy upstream field.
Click Copy Upstream Field, and the system will automatically generate output fields based on the upstream field names.
Manage output fields.
You can also perform the following operations on the added fields:
Click the Actions column's
icon to edit existing fields.Click the Actions column
icon to delete the existing field.
Mapping
The mapping relationship is used to map the input fields of the source table to the output fields of the target table, facilitating subsequent data synchronization. The mapping relationship includes same-name mapping and same-row mapping. The scenarios are described as follows:
Same-name Mapping: Maps fields with the same field name.
Same-row Mapping: The field names of the source table and the target table are inconsistent, but the data in the corresponding rows of the fields need to be mapped. Only fields in the same row are mapped.
Click Confirm to finalize the property configuration for the Hbase Output Widget.