Dataphin:Configure the GBase 8a input component

Prerequisites

• You have created a GBase 8a data source. For more information, see Create a GBase 8a data source.

• To configure the GBase 8a input component, your account needs the read permission for synchronization on the data source. If you do not have the permission, you must apply for it. For more information, see Apply for data source permissions.

Procedure

1. In the top navigation bar of the Dataphin homepage, choose Develop > data integration.

2. In the top navigation bar, select a Project. If you are using the Dev-Prod mode, you must also select an environment.

3. In the left-side navigation pane, click offline integration. In the offline integration list, click the desired offline pipeline to open its configuration page.

4. In the upper-right corner of the page, click component library to open the component library panel.

5. In the left-side navigation pane of the component library panel, select Input. From the list of input components on the right, find the GBase 8a component and drag it onto the canvas.

6. On the GBase 8a input component card, click the icon to open the GBase 8a input configuration dialog box.

7. In the GBase 8a input configuration dialog box, configure the parameters.

   Parameter	Description
   step name	The name of the GBase 8a input component. Dataphin automatically generates a step name, which you can change as needed. The name must meet the following requirements: Can contain only Chinese characters, letters, underscores (_), and digits. Cannot exceed 64 characters in length.
   data source	This drop-down list shows all GBase 8a data sources in the current Dataphin instance, regardless of whether you have the read permission for synchronization. Click the icon to copy the name of the current data source. For a data source where you lack read permission for synchronization, you can apply for it. For more information, see Apply for data source permissions. If no GBase 8a data source exists, click Create data source. For details, see Create a GBase 8a data source.
   number of source tables	Select the number of source tables. The options are single table and multiple tables. single table: Synchronizes data from a single source table to a single destination table. multiple tables: Synchronizes data from multiple source tables to a single destination table. The system uses a union operation to merge the data.
   Table matching method	Currently, only General rule is available. Note This parameter is available only when you set number of source tables to multiple tables.
   Table	Select the source table or tables. If you set number of source tables to single table, you can search for a table by entering a keyword, or enter the exact table name and click Exact search. After you select a table, the system automatically checks its status. Click the icon to copy the name of the selected table. If you set number of source tables to multiple tables, perform the following steps to add tables: In the input box, enter an expression to select tables that share the same schema. The system supports enumerated lists, regular expression-like patterns, and a combination of both. For example: table_[001-100];table_102. Click Exact search. In the Confirm match details dialog box, view the list of matching tables. Click Confirm.
   split key (Optional)	The system splits data into shards based on the specified split key. You can use this parameter with concurrency settings to enable parallel reads. For optimal performance, select a primary key or an indexed column. Important If you select a column of a date and time data type, the system identifies the minimum and maximum values and performs a rough split based on the total time range and the degree of concurrency. The resulting data shards are not guaranteed to be of equal size.
   batch size (Optional)	The number of records to read in a single batch. Reading in batches (for example, 1,024 records at a time) instead of one record at a time reduces database interactions, which improves I/O efficiency and lowers network latency.
   input filter (Optional)	A filter condition for data extraction. You can configure the filter in the following ways: Specify a static value to extract specific data. Example: ds=20210101. Use a variable to extract a specific partition of data. Example: ds=${bizdate}.
   output fields	This section displays all matching fields from the selected tables. You can perform the following operations: field management: Remove fields that you do not want to output to downstream components. Delete individual fields: Click the icon in the Actions column to remove a field. Delete fields in batches: Click field management. In the field management dialog box, select multiple fields, click the left arrow icon to move them to the unselected list, and then click OK. add fields in batches: Click add fields in batches to configure multiple fields at once using JSON, TEXT, or DDL format. Note After you add fields in batches and click OK, the new fields overwrite any existing field configuration. To configure fields in batches in JSON format, use the following example: // Example: [{ "index": 1, "name": "id", "type": "int(10)", "mapType": "Long", "comment": "comment1" }, { "index": 2, "name": "user_name", "type": "varchar(255)", "mapType": "String", "comment": "comment2" }] Note In the JSON code, index specifies the 1-based column number, name specifies the imported field name, and type specifies the imported field type. For example, "index":3,"name":"user_id","type":"String" maps the third column from the source file to a new field named user_id with a String type. To configure fields in batches in TEXT format, use the following example: // Example: 1,id,int(10),Long,comment1 2,user_name,varchar(255),Long,comment2 The row delimiter separates the information for each field. The default is the newline character (\n). Supported delimiters include the newline character (\n), semicolon (;), and period (.). A column delimiter is used to separate field names from field types. The default is a half-width comma (,).It supports','. The field type is optional, and the default is','. To configure fields in batches in DDL format, use the following example: CREATE TABLE tablename ( user_id serial, username VARCHAR(50), password VARCHAR(50), email VARCHAR (255), created_on TIMESTAMP, ); create an output field: Click +Create output field. Enter a Field name, Type, and Comment, and select a mapping type. After you configure the field, click the icon to save the row. system fields: You can select system fields to track the origin of each row of data. This is useful when you synchronize data from multiple tables, for example, from sharded databases and tables, to a single destination table. After you select a system field, the system automatically populates its field type and mapping type. dp_datasource: The name of the data source. dp_dbname: The name of the database, if any. dp_schema: The name of the schema, if any. dp_source_table: The name of the source table. dp_read_timestamp: The start time of the SELECT operation at the source. You can concatenate multiple system fields. Example: dp_dbname.dp_source_table. Note You cannot modify the field type or mapping type of system fields. The dp_read_timestamp field must be used as a standalone field and cannot be concatenated with others.

8. Click Confirm to save the GBase 8a input component configuration.