Dataphin:Configure the TiDB output component

Prerequisites

• You have created a TiDB data source. For more information, see Create a TiDB data source.

• The account used to configure the TiDB output component properties has write-through permission for the data source. If you do not have the permission, you need to request it. For more information, see Apply for, Renew, and Return Data Source Permissions.

Procedure

1. In the top navigation bar of the Dataphin homepage, choose Development > Data Integration.

2. In the top navigation bar of the integration page, select Project (In Dev-Prod mode, you need to select Environment).

3. In the left navigation pane, click Batch Pipeline, and then click the offline pipeline that you want to develop in the Batch Pipeline list to open the configuration page of the offline pipeline.

4. Click Component Library in the upper-right corner of the page to open the Component Library panel.

5. In the left navigation pane of the Component Library panel, select Outputs, find the TiDB component in the output component list on the right, and drag the component to the canvas.

6. Click and drag the icon of the target input, transform, or flow component to connect it to the current TiDB output component.

7. Click the icon on the TiDB output component to open the TiDB Output Configuration dialog box.

8. In the TiDB Output Configuration dialog box, configure the parameters.

   Parameter		Description
   Basic Settings	Step Name	The name of the TiDB output component. Dataphin automatically generates a name that you can modify. The name must meet the following requirements: It can contain only Chinese characters, letters, underscores (_), and digits. It cannot exceed 64 characters in length.
   	Datasource	The dropdown list displays all TiDB data sources, regardless of whether you have write-through permission. Click the icon to copy the data source name. For data sources for which you do not have the write-through permission, you can click Request next to the data source to request the write-through permission. For more information, see Apply for, Renew, and Return Data Source Permissions. If you do not have a TiDB data source, click Create Data Source to create one. For more information, see Create a TiDB data source.
   	Table	Select the target table for output data. You can enter a keyword to search for tables, or enter the exact table name and click Exact Match. After you select a table, the system automatically checks the table status. Click the icon to copy the name of the selected table.
   	Loading Policy	The policy for writing data to the target table. Options: Append Data (insert Into): Appends data to the existing data in the target table without modifying historical data. When a primary key or constraint violation occurs, a dirty data error is reported. Overwrite On Primary Key Conflict (replace Into): When a primary key or constraint violation occurs, the system first deletes the entire row of old data with the duplicate primary key, and then inserts the new data. Update On Primary Key Conflict (on Duplicate Key Update): When a primary key or constraint violation occurs, the system updates the data of the mapped fields on the existing record.
   	Batch Write Data Size (optional)	The maximum data size per batch write. The system flushes a batch when either this limit or Batch Write Count is reached. Default: 32M.
   	Batch Write Count (optional)	The maximum number of records per batch write. Default: 2048 records. The system flushes a batch when either Batch Write Count or Batch Write Data Size is reached. When the accumulated data reaches either limit (batch write data size or batch write count), the system immediately flushes the batch to the target. We recommend setting the batch write data size to 32 MB. Set the batch write count higher than the data-size limit divided by the average record size so that the data-size limit triggers writes first. For example, if a single record is about 1 KB and the batch write data size is 16 MB, set the batch write count to a value greater than 16384 (16 MB / 1 KB), such as 20000 records. This way, writes are triggered by the data-size limit whenever accumulated data reaches 16 MB.
   	Prepare Statement (optional)	An SQL script executed on the database before data import. For example, to ensure continuous service availability, this step creates a target table Target_A and writes data to it. After the write completes, it renames the active table Service_B to Temp_C, renames Target_A to Service_B, and then deletes Temp_C.
   	End Statement (optional)	An SQL script executed on the database after data import.
   Field Mapping	Input Fields	Lists the input fields from the upstream component.
   	Output Fields	Lists the output fields. You can perform the following operations: Field Management: Click Field Management to select output fields. Click the icon to move Selected Input Fields to Unselected Input Fields. Click the icon to move Unselected Input Fields to Selected Input Fields. Batch Add: Click Batch Add to configure in JSON, TEXT, or DDL format. Configure in JSON format, for example: // Example: [{ "name": "user_id", "type": "String" }, { "name": "user_name", "type": "String" }] Note name specifies the name of the field to import, and type specifies the data type of the field after it is imported. For example, "name":"user_id","type":"String" imports the field named user_id and sets its data type to String. Configure in TEXT format, for example: // Example: user_id,String user_name,String The row delimiter is used to separate the information of each field, with the default being a line feed (\n). It supports line feed (\n), semicolon (;), and period (.). The column delimiter is used to separate the field name and field type, with the default being a comma (,). Configure in DDL format, for example: CREATE TABLE tablename ( id INT PRIMARY KEY, name VARCHAR(50), age INT ); Create New Output Field: Click +Create New Output Field, fill in the Column and select the Type as prompted. After completing the configuration for the current row, click the icon to save.
   	Mapping	Manually map fields between the upstream input and the target table. Quick Mapping includes Same Row Mapping and Same Name Mapping. Same Name Mapping: Maps fields with the same name. Same Row Mapping: Maps fields in the same row when the field names in the source and target tables are different but the data in the corresponding rows needs to be mapped.

9. Click OK to complete the property configuration of the TiDB output component.