Dataphin:Configure PolarDB-X (formerly DRDS) Output Widget

Prerequisites

• A PolarDB-X data source has been created. For more information, see Create PolarDB-X Data Source.

• To configure the properties of the PolarDB-X output widget, the account must possess the necessary write-through permissions for the data source. Should you lack these permissions, you will need to obtain them for the data source. For more information, see Request, Renew, and Return Data Source Permissions or .

Procedure

1. On the Dataphin home page, navigate to Development > Data Integration via the top menu bar.

2. On the integration page, select Project from the top menu bar (choose Environment in Dev-Prod mode).

3. In the navigation pane on the left, click on the Batch Pipeline. From the Batch Pipeline list, select the offline pipeline you want to develop to access its configuration page.

4. To open the Component Library panel, click on the Component Library in the upper right corner of the page.

5. In the Component Library panel's left-side navigation pane, select Output. Then, locate the Polardb-x (formerly DRDS) widget in the output widget list on the right and drag it onto the canvas.

6. Connect the target input, transform, or flow widget to the PolarDB-X (formerly DRDS) output widget by clicking and dragging the icon.

7. To configure the widget, click the icon on the PolarDB-X (formerly DRDS) output widget card, which opens the Polardb-x (formerly DRDS) Output Configuration dialog box.

   

8. In the Polardb-x (formerly DRDS) Output Configuration dialog box, set the necessary parameters.

   Parameter		Description
   Basic Settings	Step Name	This is the name of the PolarDB-X output widget. Dataphin automatically generates the step name, and you can also modify it according to the business scenario. The name must meet the following requirements: Can only contain Chinese characters, letters, underscores (_), and numbers. Cannot exceed 64 characters.
   	Datasource	In the data source drop-down list, all PolarDB-X type data sources are displayed, including data sources for which you have write-through permission and those for which you do not. Click the icon to copy the current data source name. For data sources without write-through permission, you can click Request after the data source to request write-through permission. For specific operations, see Request Data Source Permissions. If you do not have a PolarDB-X type data source, click Create Data Source to create a data source. For specific operations, see Create PolarDB-X Data Source.
   	Table	Select the target table for output data. You can enter a table name keyword to search, or enter the exact table name and click Precise Search. After selecting a table, the system will automatically perform table status detection. Click the icon to copy the name of the currently selected table.
   	Loading Policy	Select the policy for writing data to the target table. Loading Policy includes: Overwrite Data means that the historical data in the target table is overwritten based on the current source table. Append Data means that data is appended to the existing data in the target table without modifying the historical data.
   	Batch Write Data Volume (optional)	The size of data written at one time. You can also set Batch Write Count. The system will write data according to the limit reached first among the two configurations. The default is 32M.
   	Batch Write Count (optional)	The default is 2048 entries. When data is synchronized and written, a batch write strategy is used. The parameters set include Batch Write Count and Batch Write Data Volume. When the accumulated data volume reaches any of the set limits (that is, the batch write data volume or count limit), the system will consider a batch of data to be full and will immediately write this batch of data to the target end at one time. It is recommended to set the batch write data volume to 32MB. For the upper limit of batch insert count, you can flexibly adjust it according to the actual size of a single record. It is usually set to a larger value to fully utilize the advantages of batch writing. For example, if the size of a single record is about 1KB, you can set the batch insert byte size to 16MB. Considering this condition, set the batch insert count to greater than the result of 16MB divided by the size of a single record 1KB (that is, greater than 16384 entries). Here, it is assumed to be set to 20000 entries. After such configuration, the system will trigger a batch write operation based on the batch insert byte size. Each time the accumulated data volume reaches 16MB, a write action will be executed.
   	Preparation Statement (optional)	The SQL script executed on the database before data import. For example, to ensure the continuous availability of the service, before the current step writes data, first create the target table Target_A, execute the write to the target table Target_A, and after the current step writes data, rename the table Service_B that continuously provides services in the database to Temp_C, then rename the table Target_A to Service_B, and finally delete Temp_C.
   	End Statement (optional)	The SQL script executed on the database after data import.
   Field Mapping	Input Field	The input fields are displayed based on the output of the upstream widget.
   	Output Field	The output fields are displayed for you. You can perform the following operations: Field Management: Click Field Management to select output fields. Click the icon to move the Selected Input Fields to Unselected Input Fields. Click the icon to move the Unselected Input Fields to Selected Input Fields. Batch Add: Click Batch Add to support batch configuration in JSON, TEXT format, or DDL format. Batch configuration in JSON format, for example: // Example: [{ "name": "user_id", "type": "String" }, { "name": "user_name", "type": "String" }] Note name specifies the name of the imported field, and type specifies the data type of the field after it is imported. For example, "name":"user_id","type":"String" imports the field named user_id and sets its data type to String. Batch configuration in TEXT format, for example: // Example: user_id,String user_name,String The row delimiter is used to separate the information of each field. The default is a line feed (\n), and it can support line feed (\n), semicolon (;), or period (.). The column delimiter is used to separate the field name and field type. The default is a comma (,). Batch configuration in DDL format, for example: CREATE TABLE tablename ( id INT PRIMARY KEY, name VARCHAR(50), age INT ); Create New Output Field: Click +create New Output Field, fill in Column and select Type according to the page prompts. After completing the configuration of the current row, click the icon to save.
   	Quick Mapping	Based on the input from upstream and the fields of the target table, you can manually select field mapping. Quick Mapping includes Row Mapping and Name Mapping. Name Mapping: Map fields with the same field name. Row Mapping: The field names of the source table and target table are inconsistent, but the data in the corresponding rows of the fields needs to be mapped. Only map fields in the same row.

9. Finalize the property configuration of the PolarDB-X output widget by clicking Confirm.