Dataphin:Configure MySQL input component

Prerequisites

• A MySQL data source is added. For more information, see Create a MySQL data source.

• The account used to configure the MySQL input component properties must have the read-through permission on the data source. If you do not have the permission, you need to request the data source permission. For more information, see Request data source permissions.

Procedure

1. In the top navigation bar of the Dataphin homepage, choose Develop > Data Integration.

2. In the top navigation bar of the Integration page, select a project (In Dev-Prod mode, you need to select an environment).

3. In the left-side navigation pane, click Batch Pipeline. In the Batch Pipeline list, click the offline pipeline that you want to develop to open its configuration page.

4. Click Component Library in the upper-right corner of the page to open the Component Library panel.

5. In the left-side navigation pane of the Component Library panel, select Inputs. Find the MySQL component in the input component list on the right, and drag it to the canvas.

6. Click the icon in the MySQL input component card to open the MySQL Input Configuration dialog box.

7. In the MySQL Input Configuration dialog box, configure the parameters.

   Parameter	Description
   Step Name	The name of the MySQL input component. Dataphin automatically generates a step name, which you can modify based on your business scenario. The name must meet the following requirements: It can contain only Chinese characters, letters, underscores (_), and digits. It cannot exceed 64 characters in length.
   Source Table Quantity	Select the source table quantity. The source table quantity includes Single Table and Multiple Tables: Single Table: This option is applicable to scenarios where business data from one table is synchronized to a target table. Multiple Tables: This option is applicable to scenarios where business data from multiple tables is synchronized to the same target table. When data from multiple tables is written to the same data table, the union algorithm is used.
   Datasource	The data source dropdown list displays all MySQL data sources in the current Dataphin instance, including data sources for which you have the read-through permission and those for which you do not have the read-through permission. If Source Table Quantity is set to Single Table, you can select only one data source. Click the icon to copy the current data source name. If Source Table Quantity is set to Multiple Tables, you can select multiple data sources in the same time zone. For data sources for which you do not have the read-through permission, you can click Request next to the data source to request the read-through permission on the data source. For more information, see Request data source permissions. If you do not have a MySQL data source, you can click Create Data Source in the dropdown list to create a data source. For more information, see Create a MySQL data source.
   Time Zone	The time zone is used to process time format data. The default value is the time zone configured in the selected data source. This parameter cannot be modified. Note For tasks created before V5.1.2, you can select Default Data Source Configuration or Channel Configuration Time Zone. The default value is Channel Configuration Time Zone. Default Data Source Configuration: the default time zone of the selected data source. Channel Configuration Time Zone: the time zone configured in Properties > Channel Configuration for the current integration task.
   Database (optional)	Select the database where the table is located. If you do not specify a database, the database specified when the data source was registered is used. If Source Table Quantity is set to Multiple Tables, you can select multiple databases. Click the icon to view all selected databases in the Database List dialog box.
   Table	Select the source table: If Source Table Quantity is set to Single Table, you can enter a keyword to search for tables or enter an exact table name and click Exact Match. After you select a table, the system automatically checks the table status. Click the icon to copy the name of the selected table. If Source Table Quantity is set to Multiple Tables, perform the following steps to add tables. In the input box, enter a table expression to filter tables with the same structure. The system supports enumeration, regular expressions, and a combination of both. For example, table_[001-100];table_102;table_abc*. Click Exact Match. In the Confirm Match Details dialog box, view the list of matched tables. Click OK.
   Shard Key (optional)	The system shards data based on the configured shard key field. You can use this parameter with the concurrent reading configuration to implement concurrent reading. You can use a column in the source data table as the shard key. We recommend that you use the primary key or a column with an index as the shard key to ensure transmission performance. Important When you select a date and time type, the system identifies the maximum and minimum values, and performs forced sharding based on the total time range and concurrency. Even distribution is not guaranteed.
   Input Filter (optional)	Configure the filter condition for data extraction. The detailed configuration is as follows: Configure a static value: Extract the corresponding data. For example, ds=20211111. Configure a variable parameter: Extract a specific part of the data. For example, ds=${bizdate}.
   Output Fields	The Output Fields section displays all fields that match the selected table and filter conditions. You can perform the following operations: Field Management: If you do not need to output certain fields to downstream components, you can delete these fields: Delete a single field: If you need to delete a small number of fields, you can click the icon in the Operation column to delete the unnecessary fields. Delete fields in batches: If you need to delete many fields, you can click Field Management. In the Field Management dialog box, select multiple fields, click the left arrow icon to move the selected input fields to the unselected input fields, and then click OK to complete the batch deletion of fields. Batch Add: Click Batch Add to configure fields in batches in JSON, TEXT, or DDL format. Note After you complete the batch addition and click OK, the added fields will overwrite the existing field information. Configure fields in batches in JSON format, for example: // Example: [{ "index": 1, "name": "id", "type": "int(10)", "mapType": "Long", "comment": "comment1" }, { "index": 2, "name": "user_name", "type": "varchar(255)", "mapType": "String", "comment": "comment2" }] Note index indicates the column number of the specified object, name indicates the field name after import, and type indicates the field type after import. For example, "index":3,"name":"user_id","type":"String" indicates that the fourth column in the file is imported, the field name is user_id, and the field type is String. Configure fields in batches in TEXT format, for example: // Example: 1,id,int(10),Long,comment1 2,user_name,varchar(255),Long,comment2 The row delimiter is used to separate the information of each field. The default value is a line feed (\n). The system supports line feeds (\n), semicolons (;), and periods (.). The column delimiter is used to separate the field name and field type. The default value is a comma (,). The system supports ','. The field type can be left empty. The default value is ','. Configure fields in batches in DDL format, for example: CREATE TABLE tablename ( user_id serial, username VARCHAR(50), password VARCHAR(50), email VARCHAR (255), created_on TIMESTAMP, ); Create an output field: Click +Create Output Field, and fill in Column, Type, and Description, and select Mapping Type as prompted. After you complete the configuration of the current row, click the icon to save it.

8. Click OK to complete the property configuration of the MySQL input component.