Dataphin:Configure the DM (DAMENG) output component

Prerequisites

• You have created a DM (DaMeng) data source. For more information, see Create a DaMeng (DM) data source.

• The account used to configure the DM (DaMeng) output component properties must have write-through permission for the data source. If you do not have the permission, you need to apply for the data source permission. For more information, see Apply for, renew, and return data source permissions.

Procedure

1. In the top navigation bar of the Dataphin homepage, choose Develop > Data Integration.

2. In the top navigation bar of the integration page, select Project (In Dev-Prod mode, you need to select Environment).

3. In the navigation pane on the left, click Batch Pipeline, and then click the offline pipeline that you want to develop in the Batch Pipeline list to open the configuration page of the offline pipeline.

4. Click Component Library in the upper-right corner of the page to open the Component Library panel.

5. In the navigation pane on the left of the Component Library panel, select Outputs, find the DM component in the output component list on the right, and drag the component to the canvas.

6. Click and drag the icon of the target input, transform, or flow component to connect it to the current DM output component.

7. On the DM output component, click the icon to open the DAMENG Output Configuration dialog box.

8. In the DaMeng Output Configuration dialog box, configure the parameters as described in the following table.

   Parameter		Description
   Basic Settings	Step Name	The name of the DM output component. Dataphin automatically generates a step name, which you can modify based on your business scenario. The name must meet the following requirements: It can contain only Chinese characters, letters, underscores (_), and digits. It cannot exceed 64 characters in length.
   	Datasource	The data source dropdown list displays all DM (DaMeng) type data sources, including those for which you have write-through permissions and those for which you do not. Click the icon to copy the current data source name. For data sources without write-through permissions, you can click Request after the data source to request write-through permissions for the data source. For more information, see Request, renew, and return data source permissions. If you do not have a DM (DaMeng) type data source, click Create Data Source to create one. For more information, see Create a DaMeng (DM) data source.
   	Table	Select the destination table for output data. You can enter a keyword to search for a table, or enter the exact table name and click Exact Match. After you select a table, the system automatically checks the table status. Click the icon to copy the name of the selected table. If the destination table for data synchronization does not exist in the DM data source, you can use the One-click Table Creation feature to quickly generate the destination table. The detailed steps are as follows: Click One-click Table Creation. Dataphin automatically matches the code to create the destination table, including the destination table name (default is the source table name), field types (initially converted based on Dataphin fields), and other information. You can modify the SQL script for creating the destination table as needed, and then click Create. After the destination table is successfully created, Dataphin automatically sets the newly created table as the destination table for output data. One-click Table Creation is used to create destination tables for data synchronization in development and production environments. Dataphin selects Create Table In Production Environment by default. If a table with the same name and structure already exists in the production environment, you do not need to select this option. Note If a table with the same name exists in the development or production environment, Dataphin will report an error indicating that the table already exists when you click Create.
   	Loading Policy	Select the strategy for writing data to the destination table. Loading Policy includes: Append Data (insert Into): When a primary key/constraint conflict occurs, a dirty data error will be reported. Update On Primary Key Conflict (merge Into): When a primary key/constraint conflict occurs, the data of the mapped fields will be updated on the existing record.
   	Synchronous Write	The primary key update syntax is not an atomic operation. If the data to be written has duplicate primary keys, you need to enable synchronous write. Otherwise, parallel write is used. Synchronous write has lower performance than parallel write. Note This option is only available when the loading policy is set to Update on Primary Key Conflict.
   	Batch Write Data Volume (optional)	The size of data to be written at once. You can also set Batch Write Count. The system will write data when either limit is reached. The default is 32M.
   	Batch Write Count (optional)	The default is 2048 records. When data is synchronized and written, a batch write strategy is used. The parameters include Batch Write Count and Batch Write Data Volume. When the accumulated data volume reaches either of the set limits (i.e., the batch write data volume or count limit), the system considers a batch of data to be full and immediately writes this batch of data to the destination at once. It is recommended to set the batch write data volume to 32MB. For the batch insert count limit, you can adjust it flexibly based on the actual size of a single record, usually setting it to a larger value to fully utilize the advantages of batch writing. For example, if the size of a single record is about 1KB, you can set the batch insert byte size to 16MB, and considering this condition, set the batch insert count to a value greater than the result of 16MB divided by the single record size of 1KB (i.e., greater than 16384 records), assuming here it is set to 20000 records. With this configuration, the system will trigger batch write operations based on the batch insert byte size, executing a write operation whenever the accumulated data volume reaches 16MB.
   	Preparation Statement (optional)	The SQL script to be executed on the database before data import. For example, to ensure continuous service availability, before the current step writes data, it first creates a target table Target_A, executes writing to Target_A, and after the current step completes writing data, it renames the continuously serving table Service_B to Temp_C, then renames table Target_A to Service_B, and finally deletes Temp_C.
   	Completion Statement (optional)	The SQL script to be executed on the database after data import.
   Field Mapping	Input Fields	Displays the input fields based on the upstream output.
   	Output Fields	Displays the output fields. You can perform the following operations: Field management: Click Field Management to select output fields. Click the icon to move Selected Input Fields to Unselected Input Fields. Click the icon to move Unselected Input Fields to Selected Input Fields. Batch add: Click Batch Add to configure in JSON, TEXT, or DDL format. Batch configuration in JSON format, for example: // Example: [{ "name": "user_id", "type": "String" }, { "name": "user_name", "type": "String" }] Note name represents the imported field name, and type represents the field type after import. For example, "name":"user_id","type":"String" means importing a field named user_id and setting its field type to String. Batch configuration in TEXT format, for example: // Example: user_id,String user_name,String The row delimiter is used to separate the information of each field, with the default being a line feed (\n). It supports line feed (\n), semicolon (;), and period (.). The column delimiter is used to separate field names from field types, with the default being a comma (,). Batch configuration in DDL format, for example: CREATE TABLE tablename ( id INT PRIMARY KEY, name VARCHAR(50), age INT ); Create new output field: Click +Create New Output Field, fill in the Column and select the Type as prompted. After completing the configuration for the current row, click the icon to save.
   	Mapping	Based on the upstream input and the fields of the target table, you can manually select field mappings. Mapping includes Same Row Mapping and Same Name Mapping. Same name mapping: Maps fields with the same name. Same row mapping: Maps data in corresponding rows when the field names in the source and target tables are inconsistent. Only maps fields in the same row.

9. Click OK to complete the property configuration of the DaMeng Output Component.