Dataphin:Configuring the openGauss output component

Prerequisites

• You have created an openGauss data source. For more information, see Create an openGauss data source.

• The account used to configure the openGauss input component properties must have the write-through permission for the data source. If you do not have the permission, you need to request the data source permission. For more information, see Request data source permissions.

Procedure

1. In the top navigation bar of the Dataphin homepage, choose Development > Data Integration.

2. In the top navigation bar of the Integration page, select Project (In Dev-Prod mode, you need to select Environment).

3. In the navigation pane on the left, click Batch Pipeline, and then click the offline pipeline that you want to develop in the Batch Pipeline list to open the configuration page of the offline pipeline.

4. Click Component Library in the upper-right corner of the page to open the Component Library panel.

5. In the navigation pane on the left of the Component Library panel, select Outputs, find the openGauss component in the output component list on the right, and drag the component to the canvas.

6. Click and drag the icon of the target input, transform, or flow component to connect it to the current openGauss output component.

7. Click the icon on the openGauss output component to open the openGauss Output Configuration dialog box.

8. In the OpenGauss Output Configuration dialog box, configure the parameters.

   Parameter		Description
   Basic settings	Step Name	The name of the openGauss output component. Dataphin automatically generates a step name, which you can modify based on your business scenario. The name must meet the following requirements: It can contain only Chinese characters, letters, underscores (_), and digits. It cannot exceed 64 characters in length.
   	Datasource	The data source dropdown list displays all openGauss data sources, including those for which you have the write-through permission and those for which you do not have the write-through permission. Click the icon to copy the current data source name. For data sources for which you do not have the write-through permission, you can click Request next to the data source to request the write-through permission for the data source. For more information, see Request data source permissions. If you do not have an openGauss data source, click Create Data Source to create a data source. For more information, see Create an openGauss data source.
   	Schema	The schema is equivalent to the project to which the table in the data source belongs.
   	Table	Select the target table for output data. Note Views are not supported in copy mode. If there is no target table for data synchronization in the openGauss data source, you can use the one-click target table generation feature to quickly generate a target table. The detailed steps are as follows: Click Generate Target Table With One Click. Dataphin automatically matches the code for creating the target table, including the target table name (default is the source table name), field types (initially converted based on Dataphin fields), and other information. The following figure shows an example: You can modify the SQL script for creating the target table as needed, and then click Create. After the target table is created, Dataphin automatically sets the newly created table as the target table for output data. The one-click target table generation feature is used to create target tables for data synchronization in the development environment and production environment. Dataphin selects the production environment for table creation by default. If a table with the same name and structure already exists in the production environment, you do not need to select table creation in the production environment. Note If a table with the same name exists in the development environment or production environment, Dataphin reports an error indicating that the table already exists when you click Create.
   	Loading Policy	Select the policy for writing data to the target table. Loading Policy includes: Overwrite Data: Overwrites the historical data in the target table based on the current source table. Append Data: Appends data to the existing data in the target table without modifying the historical data. copy: Used for copying between tables and files. When conflicts occur, the system handles them according to the Conflict Resolution Policy, either Report Error On Conflict or Overwrite On Conflict.
   	Batch write data volume	The size of data to be written at one time. You can also set Batch Write Count. When writing data, the system writes data based on whichever limit is reached first. The default value is 32M.
   	Batch write count	The default value is 2048 records. During data synchronization and writing, a batch writing strategy is used, with parameters including Batch Write Count and Batch Write Data Volume. When the accumulated data volume reaches either of the set limits (i.e., the batch write data volume or count limit), the system considers a batch of data to be full and immediately writes this batch of data to the target end at once. It is recommended to set the batch write data volume to 32MB. For the batch insert count limit, you can adjust it flexibly based on the actual size of a single record, usually setting it to a larger value to fully utilize the advantages of batch writing. For example, if the size of a single record is about 1KB, you can set the batch insert byte size to 16MB, and considering this condition, set the batch insert count to a value greater than the result of 16MB divided by the single record size of 1KB (i.e., greater than 16384 records), assuming here it is set to 20000 records. With this configuration, the system will trigger batch writing operations based on the batch insert byte size, executing a write operation whenever the accumulated data volume reaches 16MB.
   Field mapping	Input Fields	Displays the input fields based on the output of the upstream component.
   	Output Fields	Displays the output fields. You can perform the following operations: Field management: Click Field Management to select output fields. Click the icon to move Selected Input Fields to Unselected Input Fields. Click the icon to move Unselected Input Fields to Selected Input Fields. Batch add: Click Batch Add to support batch configuration in JSON, TEXT format, and DDL format. Batch configuration in JSON format, for example: // Example: [{ "name": "user_id", "type": "String" }, { "name": "user_name", "type": "String" }] Note `name` specifies the name of the field to import. `type` specifies the type of the field after it is imported. For example, "name":"user_id","type":"String" imports the field named `user_id` and sets its type to `String`. Batch configuration in TEXT format, for example: // Example: user_id,String user_name,String The row delimiter is used to separate the information of each field, with the default being a line feed (\n). It supports line feed (\n), semicolon (;), and period (.). The column delimiter is used to separate the field name and field type, with the default being a comma (,). Batch configuration in DDL format, for example: CREATE TABLE tablename ( id INT PRIMARY KEY, name VARCHAR(50), age INT ); Create new output field: Click +Create New Output Field, fill in the Column and select the Type as prompted on the page. After completing the configuration for the current row, click the icon to save.
   	Mapping	Based on the upstream input and the fields of the target table, you can manually select field mappings. Quick Mapping includes Same Row Mapping and Same Name Mapping. Same name mapping: Maps fields with the same name. Same row mapping: The field names in the source table and target table are inconsistent, but the data in the corresponding rows of the fields needs to be mapped. Only maps fields in the same row.

9. Click OK to complete the property configuration of the OpenGauss Output Component.