Dataphin:Configure the DataHub input component

Prerequisites

• You have created a DataHub data source. For more information, see Create a DataHub data source.

• Your account must have read permission for the data source. If not, apply for it. For more information, see Apply for, renew, and return data source permissions.

Procedure

1. In the top menu bar of Dataphin, choose Develop > Data Integration.

2. On the Data Integration page, select a project from the top menu bar. If you are in Dev-Prod mode, you must also select an environment.

3. In the left navigation pane, click Batch Integration. In the Batch Integration list, click the desired batch pipeline.

4. In the upper-right corner of the page, click Component Library to open the Component Library panel.

5. In the Component Library panel, select Input from the left navigation pane. Find the DataHub component and drag it onto the canvas.

6. On the DataHub input component card, click the icon to open the DataHub input configuration dialog box.

7. In the DataHub input configuration dialog box, configure the parameters as described in the following table.

   Parameter	Description
   Step name	The name of the DataHub input component. Dataphin automatically generates a step name, but you can change it to fit your business needs. The name must meet the following requirements: Can contain only Chinese characters, letters, underscores (_), and digits. Cannot exceed 64 characters in length.
   Data source	From the drop-down list, select the DataHub data source to read from. The list displays all DataHub data sources in Dataphin, regardless of your permissions. Click the icon to copy the data source name. If you lack read permission for a data source, click Apply next to its name to request access. For more information, see Apply for, renew, or revoke data source permissions. If no DataHub data source exists, click New to create one. For more information, see Create a DataHub data source.
   Topic	The name of the DataHub Topic. From the drop-down list, select the Topic. The system then automatically identifies the Topic type as either Tuple or Blob.
   Consumption start time	Specifies the start of the data consumption window. This value must be a string in the yyyyMMddHHmmss format. This parameter must be used with a scheduling parameter. For example, if your scheduling parameter is startTime=${20220101000000}, set the consumption start time to ${startTime}.
   Consumption end time	Specifies the end of the data consumption window. This value must be a string in the yyyyMMddHHmmss format. This parameter must be used with a scheduling parameter. For example, if your scheduling parameter is endTime=${20220101000000}, set the consumption end time to ${endTime}.
   Batch size	The number of records to fetch in a single batch. Reading data in batches (for example, 1,024 records at a time) instead of one record at a time reduces the number of round-trips to the data source. This improves I/O performance and lowers network latency.
   Output field	If the Topic type is Blob: The component outputs a fixed set of fields: shardId, systemTime, and Base64Data. The data in a Blob Topic is Base64-encoded into a string. If the Topic type is Tuple: The output field area displays all fields from the selected Topic. To exclude fields from being passed to downstream components, remove them: To remove a single field: Click the icon in the Actions column for that field. To remove multiple fields: Click field management. In the field management dialog box, select the fields to remove, click the left arrow icon to move them to the unselected fields list, and then click OK.

8. Click OK to save the configuration.