Dataphin:Configure GBase 8a output components

Prerequisites

• You have added a GBase 8a data source. For more information, see Create a GBase 8a data source.

• The account used to configure the GBase 8a output component properties has the write-through permission for the data source. If you do not have the permission, you need to request the data source permission. For more information, see Request, renew, and return data source permissions.

Procedure

1. In the top navigation bar of the Dataphin homepage, choose Develop > Data Integration.

2. In the top navigation bar of the integration page, select Project (In Dev-Prod mode, you need to select Environment).

3. In the left navigation pane, click Batch Pipeline, and then click the offline pipeline that you want to develop in the Batch Pipeline list to open the configuration page of the offline pipeline.

4. Click Component Library in the upper-right corner of the page to open the Component Library panel.

5. In the left navigation pane of the Component Library panel, select Outputs, find the GBase 8a component in the output component list on the right, and drag the component to the canvas.

6. Click and drag the icon of the target input, transform, or flow component to connect it to the current GBase 8a output component.

7. On the GBase 8a output component, click the icon to open the GBase 8a Output Configuration dialog box.

8. In the GBase 8a Output Configuration dialog box, configure the parameters.

   	Parameter	Description
   Basic Settings	Step Name	The name of the GBase 8a output component. Dataphin automatically generates a step name, which you can modify as needed. The name must meet the following requirements: It can contain only Chinese characters, letters, underscores (_), and digits. It cannot exceed 64 characters in length.
   	Datasource	The data source dropdown list displays all GBase 8a data sources, including those for which you have the write-through permission and those for which you do not have the write-through permission. Click the icon to copy the current data source name. For data sources for which you do not have the write-through permission, you can click Request next to the data source to request the write-through permission for the data source. For more information, see Request data source permissions. If you do not have a GBase 8a data source, click Create Data Source to create a data source. For more information, see Create a GBase 8a data source.
   	Table	Select the target table for output data. You can enter a keyword to search for a table or enter the exact table name and click Exact Match. After you select a table, the system automatically checks the table status. Click the icon to copy the name of the selected table.
   	Loading Policy	Select the policy for writing data to the target table. The loading policy supports Append Data. When a primary key or constraint violation occurs, a dirty data error is reported.
   	Batch Write Data Volume (optional)	The size of data to be written at a time. You can also set Batch Write Records. The system writes data when either of the two limits is reached. The default value is 32M.
   	Batch Write Records (optional)	The default value is 2048 records. When data is synchronized and written, the batch write strategy is used. The parameters include Batch Write Records and Batch Write Data Volume. When the accumulated data volume reaches either of the limits (the batch write data volume or batch write records), the system considers that a batch of data is accumulated and immediately writes the batch of data to the destination at a time. We recommend that you set the batch write data volume to 32 MB. For the upper limit of batch write records, you can adjust it based on the actual size of a single record. Usually, set it to a large value to fully utilize the advantages of batch writing. For example, if the size of a single record is about 1 KB, you can set the batch write data volume to 16 MB. Considering this condition, set the batch write records to a value greater than the result of 16 MB divided by the size of a single record 1 KB (that is, greater than 16,384 records), for example, 20,000 records. After this configuration, the system triggers batch writing based on the batch write data volume. When the accumulated data volume reaches 16 MB, a write operation is performed.
   	Preparation Statement (optional)	The SQL script to be executed on the database before data import. For example, to ensure the continuous availability of the service, before the current step writes data, create the target table Target_A, execute writing to the target table Target_A. After the current step completes writing data, rename the table Service_B that continuously provides services in the database to Temp_C, then rename the table Target_A to Service_B, and finally delete Temp_C.
   	End Statement (optional)	The SQL script to be executed on the database after data import.
   Field Mapping	Input Fields	The input fields are displayed based on the output of the upstream component.
   	Output Fields	The output fields are displayed. You can perform the following operations: Field management: Click Field Management to select output fields. Click the icon to move Selected Input Fields to Unselected Input Fields. Click the icon to move Unselected Input Fields to Selected Input Fields. Batch add: Click Batch Add to configure in JSON, TEXT, or DDL format in batches. Configure in JSON format in batches, for example: // Example: [{ "name": "user_id", "type": "String" }, { "name": "user_name", "type": "String" }] Note name indicates the name of the imported field, and type indicates the type of the field after import. For example, "name":"user_id","type":"String" indicates that the field named user_id is imported and the field type is set to String. Configure in TEXT format in batches, for example: // Example: user_id,String user_name,String The row delimiter is used to separate the information of each field. The default value is the line feed (\n). The line feed (\n), semicolon (;), and period (.) are supported. The column delimiter is used to separate the field name from the field type. The default value is the comma (,). Configure in DDL format in batches, for example: CREATE TABLE tablename ( id INT PRIMARY KEY, name VARCHAR(50), age INT ); Create an output field: Click +Create Output Field, fill in Column and select Type as prompted on the page. After you complete the configuration of the current row, click the icon to save it.
   	Mapping	Based on the input from the upstream and the fields of the target table, you can manually select field mapping. Mapping includes Same Row Mapping and Same Name Mapping. Same Name Mapping: Maps fields with the same name. Same Row Mapping: The field names in the source table and the target table are inconsistent, but the data in the corresponding rows of the fields needs to be mapped. Only fields in the same row are mapped.

9. Click OK to complete the property configuration of the GBase 8a Output Component.