Dataphin:Configure OceanBase Output Components

Prerequisites

• An OceanBase data source has been established. For more information, see Create an OceanBase Data Source.

• To configure the properties of the OceanBase output component, the account must have write-through permission for the data source. If you lack the necessary permissions, you must obtain them from the data source. For more information, see Request, Renew, and Return Data Source Permissions.

Procedure

1. On the Dataphin home page, navigate to the top menu bar and select Development > Data Integration.

2. At the top menu bar of the integration page, select Project (in Dev-Prod mode, choose Environment).

3. In the navigation pane on the left, click Batch Pipeline. Then, in the Batch Pipeline list, select the offline pipeline you want to develop to access its configuration page.

4. Click Component Library in the upper right corner to open the Component Library panel.

5. In the Component Library panel's left-side navigation pane, select Output. Then, in the output component list on the right, locate the OceanBase component and drag it onto the canvas.

6. Connect the target input, transform, or flow component to the OceanBase output component by clicking and dragging the icon.

7. To open the OceanBase Output Configuration dialog box, click the icon on the OceanBase output component card.

8. In the Oceanbase Output Configuration dialog box, set the necessary parameters.

   	Parameter	Description
   Basic Settings	Step Name	The name of the OceanBase output component. Dataphin automatically generates the step name, and you can also modify it according to the business scenario. The naming convention is as follows: Can only contain Chinese characters, letters, underscores (_), and numbers. Cannot exceed 64 characters.
   	Datasource	In the data source drop-down list, all OceanBase type data sources are displayed, including those for which you have write-through permission and those for which you do not. Click the icon to copy the current data source name. For data sources without write-through permission, you can click Request after the data source to apply for write-through permission. For more information, see Request Data Source Permissions. If you do not have an OceanBase type data source, click Create Data Source to create a data source. For more information, see Create an OceanBase Data Source.
   	Table	Select the target table for output data. You can enter a table name keyword to search or enter an accurate table name and then click Precise Search. After selecting a table, the system will automatically perform a table status check. Click the icon to copy the name of the currently selected table. The loading policy of OceanBase varies depending on the mode. The policies for each mode are as follows: Oracle mode only supports Append Mode. When there is a primary key or constraint violation, the system will prompt a dirty data fault. When there is no primary key or constraint violation, new data is directly appended. OceanBase's MySQL mode. Append Policy: When there is no primary key or constraint violation, new data is directly appended. Overwrite Policy: When there is a primary key or constraint violation, the new value overwrites the old value (field-level overwrite, only overwriting the written fields).
   	Batch Write Data Volume (optional)	The size of the data volume written at one time. You can also set Batch Write Count. During writing, the system will write according to the limit reached first among the two configurations, with a default of 32 MB.
   	Batch Write Count (optional)	Default is 2048 rows. During data synchronization writing, a batch writing strategy is adopted, including the parameters Batch Write Count and Batch Write Data Volume. When the accumulated data volume reaches any of the set limits (that is, the limit of batch write data volume or count), the system will consider a batch of data to be full and will immediately write this batch of data to the target end at once. It is recommended to set the batch write data volume to 32 MB. For the upper limit of batch insert count, you can adjust flexibly according to the actual size of a single record, usually setting a larger value to fully utilize the advantages of batch writing. For example, if the size of a single record is approximately 1 KB, you can set the batch insert byte size to 16 MB. Considering this condition, set the batch insert count to greater than the result of 16 MB divided by the size of a single record, 1 KB (that is, greater than 16384 rows). Here, it is assumed to be set to 20000 rows. With this configuration, the system will trigger a batch write operation based on the batch insert byte size. Whenever the accumulated data volume reaches 16 MB, a write action will be executed.
   	Prepare Statement (optional)	The SQL script executed on the database before data import. For example, to ensure the continuous availability of the service, before the current step writes data, first create the target table Target_A, execute the write to the target table Target_A. After the current step writes data, rename the table Service_B, which continuously provides services in the database, to Temp_C. Then rename the table Target_A to Service_B, and finally delete Temp_C.
   	End Statement (optional)	The SQL script executed on the database after data import.
   Field Mapping	Input Fields	The input fields are displayed based on the output of the upstream component.
   	Output Fields	The output fields are displayed. You can perform the following operations: Field Management: Click Field Management to select output fields. Click the icon to move Selected Input Fields to Unselected Input Fields. Click the icon to move Unselected Input Fields to Selected Input Fields. Batch addition: Click Batch Addition. JSON, TEXT format, and DDL format batch configuration are supported. Batch configuration in JSON format, for example: // Example: [{ "name": "user_id", "type": "String" }, { "name": "user_name", "type": "String" }] Note `name` specifies the name of the imported field, and `type` specifies the data type of the field after it is imported. For example, "name":"user_id","type":"String" imports the field named `user_id` and sets its data type to String. Batch configuration in TEXT format, for example: // Example: user_id,String user_name,String The row delimiter is used to separate each field's information. The default is a line feed (\n), supporting line feed (\n), semicolon (;), or period (.). The column delimiter is used to separate the field name and field type, with the default being a comma (,). Batch configuration in DDL format, for example: CREATE TABLE tablename ( id INT PRIMARY KEY, name VARCHAR(50), age INT ); Create Output Field: Click +create Output Field, fill in Column and select Type according to the page prompts. After completing the configuration of the current row, click the icon to save.
   	Mapping	Based on the input from upstream and the fields of the target table, you can manually select field mapping. Mapping includes Row Mapping and Same Name Mapping. Same Name Mapping: Map fields with the same field name. Row Mapping: The field names of the source table and target table are inconsistent, but the data in the corresponding rows of the fields needs to be mapped. Only map fields in the same row.

9. Click Confirm to finalize the property configuration for the OceanBase Output Component.