You can configure the SelectDB output component to write data from external databases to SelectDB, or to replicate and push data from storage systems integrated with big data platforms to SelectDB for data integration and reprocessing.
Prerequisites
You have created a SelectDB data source. For more information, see Create a SelectDB data source.
The account configuring the SelectDB output component properties must have write-through permissions for the data source. If you do not have permissions, request data source permissions. For more information, see Request data source permissions.
Procedure
On the Dataphin home page, in the top menu bar, choose Development > Data Integration.
On the Integration page, in the top menu bar, select Project (In Dev-Prod mode, select an environment).
In the navigation pane on the left, click Offline Integration. Then, in the Offline Integration list, click the target offline pipeline to open its configuration page.
Click Component Library in the upper-right corner of the page to open the Component Library panel.
In the navigation pane on the left of the Component Library panel, select Outputs. In the output component list on the right, find the SelectDB component and drag it to the canvas.
Click and drag the
icon of the target input, transform, or flow component to connect it to the current SelectDB output component.Click the
icon in the SelectDB output component card to open the SelectDB Output Configuration dialog box.
In the SelectDB Output Configuration dialog box, configure the following parameters.
Parameter
Description
Basic Settings
Step Name
The name of the SelectDB output component. Dataphin automatically generates the step name. You can modify it as needed. Naming conventions are as follows:
Can contain only Chinese characters, letters, underscores (_), and numbers.
Length cannot exceed 64 characters.
Datasource
The data source drop-down list displays all SelectDB data sources, including those for which you have write-through permissions and those for which you do not. Click the
icon to copy the current data source name.For data sources that do not have write-through permissions, you can click Request next to the data source to request the permissions. For more information, see Request data source permissions.
If you do not have a SelectDB data source, click Create Data Source to create one. For detailed steps, see Create a SelectDB Data Source.
Table
Select the destination table for the output data. You can enter a table name keyword to search, or enter the exact table name and click Precise Search. After you select a table, the system automatically checks the table status. Click the
icon to copy the name of the currently selected table.If the SelectDB data source does not have a target table for data synchronization, use the one-click table creation feature to quickly generate a target table. The detailed procedure is as follows:
Click One-Click Table Creation. Dataphin automatically generates the code for creating the target table, including the target table name (which defaults to the source table name) and field types (with preliminary conversion based on Dataphin fields).
You can modify the SQL script for creating the target table as needed, and then click New. After the target table is successfully created, Dataphin automatically sets the new target table as the target table for output data.
NoteIf a table with the same name exists in the development environment, Dataphin reports an error indicating the table already exists after you click Create.
If there are no matching items, you can also integrate based on a manually entered table name.
Production Table Missing Strategy
The policy for handling cases where the production table does not exist. You can select Take No Action or Automatic Creation. The default is Automatic Creation. If you select Take no action, the production table is not created when the task is published. If you select Automatic creation, a table with the same name is created in the target environment when the task is published.
Do Not Process: If the target table does not exist, a prompt indicates that the target table does not exist upon submission, but the task can still be published. In this case, you must manually create the target table in the production environment before the task can be executed.
Automatic Creation: You must Edit The Table Creation Statement. By default, the statement of the selected table is filled in, and you can modify it. The table name in the statement uses the placeholder
${table_name}. Only this placeholder is supported, and it is replaced with the actual table name during execution.If the target table does not exist, the system first creates the table according to the table creation statement. If table creation fails, the check result is 'failed' during publishing. You can modify the table creation statement based on the error message and then publish again after modification. If the target table already exists, table creation is not performed.
NoteThis option is supported only in Dev-Prod mode projects.
Data Format
You can select CSV or JSON.
If you choose CSV, also configure CSV Import Column Delimiter and CSV Import Row Delimiter.
CSV Import Column Delimiter (Optional)
When using StreamLoad CSV import, configure the CSV import column delimiter here. The default is
_@dp@_. Do not explicitly specify it here if you use the default value. If your data contains_@dp@_, customize and use another character as the delimiter.CSV Import Row Delimiter (Optional)
When using StreamLoad CSV import, configure the CSV import row delimiter here. The default is
_#dp#_. Do not explicitly specify it here if you use the default value. If your data contains_#dp#_, customize and use another character as the delimiter.Batch Write Data Volume (Optional)
The volume of data to write in a single batch. You can set this parameter along with Batch Write Count. The system initiates the write operation when the limit of either setting is reached first. The default value is 32 MB.
Batch Write Record Count (Optional)
The default value is 2048 records. Data synchronization uses a batch writing policy. The parameters for this policy include Number Of Records Per Batch and Data Volume Per Batch.
When the accumulated data volume reaches either of the set limits (i.e., the batch write data volume or record count limit), the system considers a batch of data full and immediately writes this batch of data to the destination.
Set the batch write data volume to 32 MB. For the upper limit of batch insert record count, adjust it flexibly based on the actual size of a single record. Typically, set a larger value to fully leverage the benefits of batch writing. For example, if a single record is approximately 1 KB, set the batch insert byte size to 16 MB. Considering this condition, set the batch insert record count to a value greater than 16 MB divided by 1 KB (i.e., greater than 16384 records). Here, assume it is set to 20000 records. With this configuration, the system triggers batch write operations based on the batch insert byte size. Each time the accumulated data volume reaches 16 MB, a write action is performed.
Preparation Statement (Optional)
The SQL script executed on the database before data import.
For example, to ensure continuous service availability, before the current step writes data, create target table Target_A, write data to Target_A. After the current step finishes writing data, rename Service_B (the table continuously providing service in the database) to Temp_C, then rename Target_A to Service_B, and finally delete Temp_C.
Completion Statement (Optional)
The SQL script executed on the database after data import.
Field Mapping
Input Fields
Input fields are displayed based on the output of upstream components.
Output Fields
Output fields are displayed. The following operations are supported:
Field Management: Click Field Management to select output fields.
Click the
icon to move Selected Input Fields to Unselected Input Fields.Click the
icon to move fields from Unselected Input Fields to Selected Input Fields.
Batch Add: Click Batch Add. This supports batch configuration in JSON, TEXT, and DDL formats.
Batch configure in JSON format. For example:
// Example: [{ "name": "user_id", "type": "String" }, { "name": "user_name", "type": "String" }]NoteName indicates the imported field name, and type indicates the field type after import. For example,
"name":"user_id","type":"String"means to import the field named user_id and set its field type to String.Configure fields in batches in TEXT format, for example:
// Example: user_id,String user_name,StringRow delimiters separate information for each field. The default is line feed (\n). It supports line feed (\n), semicolon (;), and period (.).
Column delimiters separate field names and field types. The default is comma (,).
Batch configure in DDL format. For example:
CREATE TABLE tablename ( id INT PRIMARY KEY, name VARCHAR(50), age INT );
Create Output Field: Click + Create Output Field, fill in Column and select Type according to the prompts on the page. After you configure the current row, click the
icon to save.
Mapping
Mapping relationships map input fields from the source table to output fields in the target table. Mapping relationships include Same-name Mapping and Row-wise Mapping. Scenarios:
Same Name Mapping: Maps fields with the same field name.
Same Row Mapping: The field names of the source table and target table are inconsistent, but the data in corresponding rows needs to be mapped. Only fields in the same row are mapped.
Click Confirm to complete the property configuration for the SelectDB Output Component.