Dataphin:Configure Log Service input components

Prerequisites

• A Log Service data source is created. For more information, see Create a Log Service data source.

• The account used to configure the Log Service input component properties must have read-through permission on the data source. If you do not have the permission, you need to request the data source permission. For more information, see Request, renew, and return data source permissions.

Procedure

1. In the top navigation bar of the Dataphin homepage, choose Develop > Data Integration.

2. In the top navigation bar of the integration page, select a project (In Dev-Prod mode, you need to select an environment).

3. In the left-side navigation pane, click Batch Pipeline. In the Batch Pipeline list, click the offline pipeline that you want to develop to open its configuration page.

4. Click Component Library in the upper-right corner of the page to open the Component Library panel.

5. In the left-side navigation pane of the Component Library panel, select Inputs. Find the Log Service component in the input component list on the right and drag it to the canvas.

6. Click the icon in the Log Service input component card to open the Log Service Input Configuration dialog box.

7. In the Log Service Input Configuration dialog box, configure the parameters.

   Parameter	Description
   Step Name	The name of the Log Service input component. Dataphin automatically generates a step name, which you can modify based on your business scenario. The name must meet the following requirements: It can contain only Chinese characters, letters, underscores (_), and digits. It cannot exceed 64 characters in length.
   Datasource	The data source dropdown list displays all Log Service data sources, including those for which you have read-through permission and those for which you do not. Click the icon to copy the current data source name. For data sources for which you do not have read-through permission, you can click Request next to the data source to request read-through permission. For more information, see Request, renew, and return data source permissions. If you do not have a Log Service data source, click Create to create one. For more information, see Create a Log Service data source.
   Table	You can enter a keyword to search for a table, or enter the exact table name and click Exact Match. Click the icon to copy the name of the selected table. Click the icon to copy the name of the selected table.
   Log Start Time	Enter the start time for reading logs. The time format is yyyyMMddHHmmss.
   Log End Time	Enter the end time for reading logs. The time format is yyyyMMddHHmmss.
   Batch Read Count	The number of records to read at a time. When reading data from the source database, you can configure a specific batch read count (such as 1,024 records) instead of reading records one by one. This reduces the number of interactions with the data source, improves I/O efficiency, and reduces network latency.
   Output Fields	Displays the output fields. Batch Add Fields. Click Batch Add. Configure in JSON format. The following sample code provides an example: // Example: [{ "index": 0, "name": "user_id", "type": "String" }, { "index": 1, "name": "user_name", "type": "String" }] Note index indicates the column number of the specified object, name indicates the field name after import, and type indicates the field type after import. For example, "index":3,"name":"user_id","type":"String" indicates that the fourth column in the file is imported, the field name is user_id, and the field type is String. Configure in TEXT format. The following sample code provides an example: // Example: 0,user_id,String 1,user_name,String The row delimiter is used to separate the information of each field. The default is a line feed (\n). Line feed (\n), semicolon (;), and period (.) are supported. The column delimiter is used to separate the field name and field type. The default is a comma (,). Click OK. Create An Output Field. Click Create Output Field, enter the Source Index and Column, and select the Type for the field. Click a blank area to complete the field creation. Manage output fields. You can perform the following operations on added fields: Click and drag the icon next to position to change the position of the field. Click the Operation icon in the column to edit an existing field. Click the Operation icon in the column to delete an existing field.

8. Click OK to complete the property configuration of the Log Service input component.