All Products
Search

This Product

    Dataphin:Configure AnalyticDB for PostgreSQL input component

    Document Center

    Dataphin:Configure AnalyticDB for PostgreSQL input component

    Last Updated:Nov 03, 2026

    To synchronize data to a destination, first configure the input component to read from the source, and then configure the destination data source. This topic describes how to configure the AnalyticDB for PostgreSQL input component.

    Prerequisites

    Procedure

    1. On the Dataphin homepage, in the top menu bar, choose develop > data integration.

    2. In the top menu bar of the integration page, select a project. (In Dev-Prod mode, you also need to select an environment.)

    3. In the left navigation pane, click Offline Integration. In the offline integration list, click the offline pipeline you want to develop.

    4. In the upper-right corner of the page, click component library to open the component library panel.

    5. In the component library panel, select Input from the left navigation pane. Find the AnalyticDB for PostgreSQL component in the list on the right and drag it onto the canvas.

    6. Click the image icon on the AnalyticDB for PostgreSQL input component card to open the AnalyticDB for PostgreSQL input configuration dialog box.

    7. In the AnalyticDB for PostgreSQL input configuration dialog box, configure the following parameters.

      Parameter

      Description

      Step name

      The name of the AnalyticDB for PostgreSQL input component. Dataphin automatically generates a step name, which you can modify to suit your business needs. The name must follow these rules:

      • Can contain only Chinese characters, letters, digits, and underscores (_).

      • Cannot exceed 64 characters in length.

      Data source

      This drop-down list displays all AnalyticDB for PostgreSQL data sources in the current Dataphin project, even if you do not have sync read permission. Click the image icon to copy the current data source name.

      If you do not have the sync read permission for a data source, click Apply next to the data source to request the permission. For more information, see Apply for data source permissions.

      If you do not have an AnalyticDB for PostgreSQL data source, click Create Data Source to create one. For more information, see Create an AnalyticDB for PostgreSQL data source.

      Time zone

      Specifies the time zone for processing time-formatted data. By default, this parameter is set to the time zone of the selected data source and cannot be changed.

      Note

      For tasks created in versions earlier than V5.1.2, you can select either data source default configuration or channel configured time zone. The default option is channel configured time zone.

      • Data source default configuration: The default time zone of the selected data source.

      • Channel configured time zone: Uses the time zone set for the current integration task under Properties > channel configuration.

      Schema (Optional)

      Select the schema that contains the table. If you do not specify a schema, the default schema for the data source is used.

      Number of source tables

      Select the number of source tables. Options include single table and multiple tables:

      • Single table: Use this option to synchronize data from one source table to one destination table.

      • Multiple tables: Use this option to synchronize data from multiple source tables to a single destination table by using the UNION algorithm.

      Table matching mode

      You can select either general rule or database regular expression.

      Note

      This parameter is available only when Number of source tables is set to Multiple tables.

      Table

      Select the source table or tables:

      • If you set Number of source tables to single table, you can search for a table by keyword, or enter the exact table name and click exact match lookup. After a table is selected, the system automatically checks its status. Click the image icon to copy the name of the selected table.

      • If you set Number of source tables to multiple tables, enter an expression to add tables based on the selected table matching mode.

        • If the table matching mode is general rule: Enter an expression to match tables with the same structure. The system supports enumerations, simple patterns, and a mix of both. For example, table_[001-100];table_102;.

        • If the table matching mode is database regular expression: Enter a regular expression that is compatible with the source database. At runtime, the task dynamically matches tables based on this expression to define the synchronization scope.

        After you enter the expression, click exact match lookup to view a list of matched tables in the Confirm Match Details dialog box.

      Split key (Optional)

      The system shards data based on the specified split key field. This allows for parallel reads when used with the concurrency setting. For optimal performance, use the primary key or an indexed column from the source table as the split key.

      Important

      If you select a field of a date or time type, the system identifies its maximum and minimum values and then splits the data based on the total time range and concurrency level. This process does not guarantee an even distribution.

      Fetch size (Optional)

      The number of rows read from the source database in a single batch. Setting a higher fetch size reduces interactions with the data source, which improves I/O efficiency and reduces network latency.

      Filter (Optional)

      Specify the filter conditions for data extraction. Examples are as follows:

      • Use a fixed value to extract specific data, for example, ds=20210101.

      • Use a variable to extract a portion of the data, for example, ds=${bizdate}.

      Output fields
      Note

      When the tenant's compute engine is AnalyticDB for PostgreSQL, the output fields section supports viewing data classification and grading information for each field. This feature is unavailable for other compute engines.

      The output fields section displays all fields from the selected tables that match the filter criteria. You can perform the following operations:

      • Field management: If you do not need to output certain fields to downstream components, you can remove them:

        • To delete a single field: Click the sgaga icon in the Actions column for any field you want to remove.

        • To delete fields in bulk: To remove a large number of fields, click field management. In the field management dialog box, select the fields that you want to remove, click the image move-left icon to move them to the unselected list, and then click OK.

          image..png

      • Batch add: Click batch add to configure fields in bulk by using the JSON, TEXT, or DDL format.

        Note

        After you add fields in bulk and click OK, this action overwrites the existing field configuration.

        • To configure fields in bulk by using the JSON format, for example:

          // Example:
  [{
     "index": 1,
     "name": "id",
     "type": "int(10)",
     "mapType": "Long",
     "comment": "comment1"
   },
   {
     "index": 2,
     "name": "user_name",
     "type": "varchar(255)",
     "mapType": "String",
     "comment": "comment2"
 }]
          Note

          The index parameter specifies the column number, name specifies the field name after import, and type specifies the field type after import. For example, "index":3,"name":"user_id","type":"String" means that the third column in the file is imported with the field name user_id and the field type String.

        • To configure fields in bulk by using the TEXT format, for example:

          // Example:
1,id,int(10),Long,comment1
2,user_name,varchar(255),Long,comment2

          • The row delimiter separates the information for each field. The default is a newline character (\n). Semicolons (;) and periods (.) are also supported.

          • The column delimiter separates properties for each field; the default is a comma (,). The field type is optional and defaults to String if omitted.

        • To configure fields in bulk by using the DDL format, for example:

          CREATE TABLE tablename (
	user_id serial,
	username VARCHAR(50),
	password VARCHAR(50),
	email VARCHAR (255),
	created_on TIMESTAMP,
);

      • Add output field: Click + add output field, then enter the field, type, and comment, and select a mapping type. After you configure the row, click the image icon to save.

        System fields: You can select system fields to identify the source of each data row. This is useful when you synchronize data from multiple tables, such as sharded tables, to a single destination table. After you select a system field, the system automatically populates the field type and mapping type. The system fields include:

        • dp_datasource: The name of the data source being synchronized.

        • dp_dbname: The name of the database being synchronized, if applicable.

        • dp_schema: The name of the schema being synchronized, if applicable.

        • dp_source_table: The name of the source table being synchronized.

        • dp_read_timestamp: The start time of the SELECT operation at the source.

        System fields support concatenating multiple system variables, such as dp_dbname.dp_source_table.

        Note

        • The field type and mapping type of system fields cannot be modified.

        • dp_read_timestamp supports only individual configuration and does not support concatenation.

    8. Click OK to complete the configuration of the AnalyticDB for PostgreSQL input component.