Dataphin:Configure the OceanBase input component

Prerequisites

• An OceanBase data source has been created. For more information, see Create an OceanBase data source.

• The account used for configuration requires sync read permission for the data source. If not, you must apply for it. For more information, see Apply for, renew, and release data source permissions.

Procedure

1. In the top navigation bar of the Dataphin homepage, choose Development > Data Integration.

2. In the top navigation bar of the integration page, select a project. In Dev-Prod mode, you must also select an environment.

3. In the left-side navigation pane, click Offline Integration. In the Offline Integration list, click the offline pipeline that you want to develop.

4. Click Component Library in the upper-right corner of the page to open the Component Library panel.

5. In the left-side navigation pane of the component library panel, select Input. From the list of input components on the right, find the OceanBase component and drag it to the canvas.

6. Click the icon on the OceanBase input component card to open the OceanBase Input Configuration dialog box.

7. In the OceanBase Input Configuration dialog box, configure the parameters.

   Parameter	Description
   Step name	The name of the OceanBase input component. Dataphin provides a default name, which you can change. The naming rules are: Can contain only Chinese characters, letters, digits, and underscores (_). Cannot exceed 64 characters.
   Data source	The OceanBase data source to read from. The drop-down list displays all OceanBase data sources in the current Dataphin instance, regardless of your permissions. Click the icon to copy the current data source name. If you lack sync read permission for a data source, click Apply next to the data source to request the permission. For more information, see Apply for, renew, and release data source permissions. If no OceanBase data source is available, click New to create one. For more information, see Create an OceanBase data source.
   Table	The source table for data synchronization. You can search for the table by keyword, or enter the full name and click Exact Search. After you select a table, the system automatically checks its status. Click the icon to copy the name of the selected table.
   Partition key (Optional)	The column used to partition source data for parallel reads. For best performance, use a primary key or an indexed column. Important If you use a date or time column as the partition key, Dataphin finds the minimum and maximum values and splits the data based on the total time range and the parallelism setting. This split is not guaranteed to be balanced.
   Batch size (Optional)	The number of records to read in a single batch. Reading in batches, instead of one record at a time, reduces network round-trips to the data source, improving I/O efficiency and lowering network latency.
   Input filter (Optional)	A WHERE clause to filter the data read from the source. You can configure the condition in the following ways: Use a fixed value: Extracts specific data. Example: ds=20210101. Use a variable: Extracts a specific portion of data. Example: ds=${bizdate}.
   Output fields	This section displays all fields from the selected table that match the input filter. You can perform the following operations: Field management: You can remove fields that you do not want to output to downstream components: To remove a single field: Click the icon in the field's row. To remove fields in bulk: Click Field Management. In the Field Management dialog box, select the fields to remove, click the icon to move them to the unselected list, and then click OK. Batch add: Click Batch Add to configure fields in bulk by using JSON, TEXT, or DDL format. To configure fields in bulk by using the JSON format, see the following example: // Example: [{ "index": 0, "name": "id", "type": "int(10)", "mapType": "Long", "comment": "comment1" }, { "index": 1, "name": "user_name", "type": "varchar(255)", "mapType": "String", "comment": "comment2" }] Note index specifies the column number, name specifies the field name after import, and type specifies the field type after import. For example, "index":3,"name":"user_id","type":"String" imports the fourth column from the source, names the field user_id, and sets its type to String. To configure fields in bulk by using the TEXT format, see the following example: // Example: 0,id,int(10),Long,comment1 1,user_name,varchar(255),Long,comment2 The row delimiter separates each field definition. The default is a newline character (\n). Other supported delimiters include the semicolon (;) and the period (.). The column delimiter separates the field name and field type. The default is a comma (,). To configure fields in bulk by using the DDL format, see the following example: CREATE TABLE tablename ( user_id serial, username VARCHAR(50), password VARCHAR(50), email VARCHAR (255), created_on TIMESTAMP, ); Add output field: Click + Add Output Field and specify the Field, Type, and Comment, and select a Mapping Type. System fields: You can add system fields to track the origin of each data row. When you add a system field, Dataphin automatically sets its field type and mapping type. This is useful when you synchronize data from multiple tables, such as sharded databases and tables, to a single destination table. Available system fields include: dp_datasource: The name of the source data source. dp_dbname: The name of the source database, if any. dp_schema: The name of the source schema, if any. dp_source_table: The name of the source table. dp_read_timestamp: The start time of the SELECT operation at the source. You can concatenate multiple system variables, for example, dp_dbname.dp_source_table. Note The field type and mapping type of system fields cannot be modified. The dp_read_timestamp system field can be used only as a standalone field and cannot be concatenated. Note Field types in pipeline development are automatically mapped by Dataphin based on the source fields. For fields that you add manually, set the field type in the Type(Length) format, such as int(10). If the selected data source is connected to the Dataphin cluster over an internal network, you can only delete fields in the Output Fields section. You cannot edit them.

8. Click Confirm to save the OceanBase input component configuration.