All Products
Search

This Product

    Dataphin:Configure SelectDB input component

    Document Center

    Dataphin:Configure SelectDB input component

    Last Updated:Nov 03, 2026

    The SelectDB input component reads data from a SelectDB data source. To synchronize this data to other sources, first configure this component to specify the source data, then configure the destination data source.

    Prerequisites

    Procedure

    1. In the top menu bar of the Dataphin homepage, choose Develop > Data Integration.

    2. In the top menu bar of the Data Integration page, select a project. If you are in Dev-Prod mode, you must also select an environment.

    3. In the left-side navigation pane, click Offline Integration. In the Offline Integration list, click the offline pipeline that you want to configure.

    4. In the upper-right corner of the page, click Component Library to open the component library panel.

    5. In the left-side navigation pane of the component library panel, select Input. In the input component list on the right, find the SelectDB component and drag it to the canvas.

    6. Click the image icon on the SelectDB input component card to open the SelectDB Input Configuration dialog box.

    7. In the SelectDB Input Configuration dialog box, configure the parameters.

      Parameter

      Description

      step name

      The name of the SelectDB input component. Dataphin generates a default step name that you can customize.

      • Can contain only Chinese characters, letters, underscores (_), and digits.

      • Cannot exceed 64 characters in length.

      data source

      In the data source drop-down list, select a SelectDB data source. The list displays all SelectDB data sources in the current Dataphin instance, regardless of whether you have read permission.

      number of source tables

      Select whether to read from a single table or multiple tables.

      • single table: Synchronizes data from a single source table to a single destination table.

      • multiple tables: Synchronizes data from multiple source tables into a single destination table. Data from multiple tables is combined by using a UNION operation.

      table matching method

      You can select general rule or database regex.

      Note

      This option is supported only when the number of source tables is set to multiple tables.

      table

      Select the source table or tables.

      • If number of source tables is set to single table, search for a table by keyword, or enter the exact table name and click Exact Search. After you select a table, the system automatically checks its status. Click the image icon to copy the name of the selected table.

      • If number of source tables is set to multiple tables, enter an expression to add tables based on the selected table matching method.

        • If you select general rule as the table matching method, enter a table expression in the input box to filter tables with the same structure. The system supports enumeration, regular expression-like formats, and a mix of both. For example, table_[001-100];table_102;.

        • Select Database Regex as the table matching method. In the input box, enter a regular expression that is supported by the current database. The system uses this expression to match tables in the target database. When the task runs, it dynamically matches and synchronizes new tables that match the expression.

        After you enter the expression, click Exact Search to view a list of matched tables in the Confirm Match Details dialog box.

      split key (Optional)

      Specifies a column to partition data for parallel reads. Using a split key can significantly improve read performance for large tables.

      batch size (Optional)

      Specifies the number of records to read per batch. Reading in batches (for example, 1024) instead of one by one reduces interactions with the data source, improving I/O efficiency and reducing network latency.

      input filter (Optional)

      Specifies a filter condition to extract data. The following examples show how to configure the condition:

      • Use a fixed value to extract specific data. Example: ds=20210101.

      • Use a variable to extract a subset of data. Example: ds=${bizdate}.

      output field

      This section displays all fields from the selected tables that match the filter conditions. You can manage these fields in the following ways:

      • field management: If you do not need to output certain fields to downstream components, you can remove them.

        • Delete a single field: To delete a few fields, click the sgaga icon in the Actions column for each field you want to remove.

        • Delete fields in batches: To delete multiple fields at once, click Field Management. In the Field Management dialog box, select the fields to remove, click the image left arrow icon to move them to the unselected list, and then click OK.

          image..png

      • batch add: Click Batch Add to configure fields in bulk by using the JSON, TEXT, or DDL format.

        Note

        After you click OK, the batch add operation overwrites any existing field configurations.

        • To configure fields in batches by using the JSON format, see the following example:

          // Example:
  [{
     "index": 1,
     "name": "id",
     "type": "int(10)",
     "mapType": "Long",
     "comment": "comment1"
   },
   {
     "index": 2,
     "name": "user_name",
     "type": "varchar(255)",
     "mapType": "String",
     "comment": "comment2"
 }]
          Note

          The index parameter specifies the column number, name specifies the output field name, and type specifies the output field type. For example, "index":3,"name":"user_id","type":"String" specifies that the third column from the source is imported, named user_id, and set to the String type.

        • To configure fields in batches by using the TEXT format, see the following example:

          // Example:
1,id,int(10),Long,comment1
2,user_name,varchar(255),Long,comment2

          • The row delimiter separates the information for each field. The default is a newline character (\n). Semicolons (;) and periods (.) are also supported.

          • The column delimiter separates the field name and field type. The field type is optional.

        • To configure fields in batches by using the DDL format, see the following example:

          CREATE TABLE tablename (
	user_id serial,
	username VARCHAR(50),
	password VARCHAR(50),
	email VARCHAR (255),
	created_on TIMESTAMP,
);

      • new output field: Click + New Output Field and enter the field name, type, mapping type, and comment. After you complete the configuration for the current row, click the image icon to save.

        system field: You can select a system field to identify the source of each data row. This is useful when you synchronize data from multiple tables, such as sharded tables, to a single destination. When you select a system field, its type and mapping type are automatically populated. Available system fields include:

        • dp_datasource: The name of the data source being synchronized.

        • dp_dbname: The name of the database being synchronized, if applicable.

        • dp_schema: The name of the schema being synchronized, if applicable.

        • dp_source_table: The name of the source table.

        • dp_read_timestamp: The timestamp when the source SELECT query began.

        You can concatenate multiple system variables. For example, dp_dbname.dp_source_table.

        Note

        • You cannot modify the field type or mapping type of a system field.

        • The dp_read_timestamp field must be used alone and cannot be concatenated with other variables.

    8. Click OK to save the configuration for the SelectDB input component.