Dataphin:Configure HDFS Output Widget

Prerequisites

• An HDFS data source has been created. For more information, see Create an HDFS Data Source.

• To configure the properties of the HDFS input widget, the account must have write-through permission for the data source. If permission is lacking, you need to request access to the data source. For more information, see Apply for, Renew, and Return Data Source Permissions.

Procedure

1. On the Dataphin home page, select Development > Data Integration from the top menu bar.

2. In the integration page's top menu bar, select Project (Dev-Prod mode requires selecting Environment).

3. In the navigation pane on the left, click Batch Pipeline. In the Batch Pipeline list, click the offline pipeline you want to develop to open its configuration page.

4. Click the Component Library in the upper right corner to open the Component Library panel.

5. In the Component Library panel's left-side navigation pane, select Output. Find the HDFS component in the list of output widgets on the right and drag it to the canvas.

6. Click and drag the icon from the target upstream component to connect it to the HDFS output widget.

7. Click the icon on the HDFS output component to open the HDFS Output Configuration dialog box.

8. In the HDFS Output Configuration dialog box, set the parameters.

   Parameter	Description
   Step Name	The name of the HDFS output widget. Dataphin automatically generates a default name, which you can modify. The naming conventions are as follows: Can only contain Chinese characters, letters, underscores (_), and numbers. Cannot exceed 64 characters.
   Data Source	The drop-down list shows all HDFS-type data sources, including those you have write-through permission for and those you do not. Click the icon to copy the data source name. For data sources without write-through permission, click Request next to the data source to request permission. For more information, see Apply for, Renew, and Return Data Source Permissions. If you do not have an HDFS-type data source, click Create Data Source to create one. For more information, see Create an HDFS Data Source.
   File Path	Enter the absolute path of the file. The NameNode is already configured in the data source, so you do not need to add the hdfs://{namenode}:{port} prefix. For example: /hadoop/input/file.txt. The system accesses the path: hdfs://{NameNode configured for the data source}:{IPC Port configured for the data source}{entered file path}.
   File Type	Select the file type for storing the data. File Type includes Text, ORC, and Parquet.
   File Encoding	Select the file encoding. File Encoding includes UTF-8 and GBK.
   Load Policy	The policy for writing data to the target HDFS data source. Load policies include overwriting data and appending data: Overwrite Data: Deletes existing files in the target directory, then writes new data files. Append Data: Adds new data files directly to the target directory.
   Row Delimiter	For the file type Text, configure the row delimiter. Optional. If left empty, the system uses \n as the default delimiter.
   Field Delimiter	For the file type Text, configure the field delimiter. Optional. If left empty, the system uses a comma (,) as the default delimiter.
   Mark Completion File	Configure the path and format of the completion marker file. Two marking levels are supported: task-level and file-level. Task-level: Example: /ftpuser/test/SUCCESS. After the task is complete, a single empty file named SUCCESS is generated. File-level: Use an asterisk (*) as a placeholder for the data file name. Example: /ftpuser/test/.flg. An empty .flg file with the same name is generated for each data file.
   Merge Policy	Select the output threading strategy: Merge: Merges all data into one file using a single thread. Output speed may be slower for large files. Important Merge does not support appending data. Do Not Merge: Uses multi-threaded output, generating multiple files.
   Export Compressed File	Specifies whether to compress the output file. Options: Do Not Compress, Export Directly in the Selected File Type, gzip compression format, or zip compression format.
   Export Column Header	Specifies whether to include the column header in the output: Export: The field name is output in the first line of each file. Do Not Export: The first line of the file is data.
   Input Fields	Displays the output fields of the upstream component.
   Output Fields	Displays the output fields. You can configure output fields by using Batch Add or Create New Output Field: Batch Add: Click Batch Add. JSON and TEXT formats are supported for batch configuration. Batch configuration in JSON format, for example: // Example: [{"name": "user_id","type": "String"}, {"name": "user_name","type": "String"}] Note name specifies the field name and type specifies the field data type. For example, "name":"user_id","type":"String" imports the user_id field and sets its data type to String. Batch configuration in TEXT format, for example: // Example: user_id,String user_name,String The row delimiter is used to separate the information of each field. The default is a line feed (\n). Line feed (\n), semicolon (;), and period (.) are supported. The column delimiter is used to separate the field name and field type. The default is a comma (,). Create New Output Field. Click +create New Output Field and fill in Column and select Type according to the page prompts. Copy Ancestor Table Field. Click Copy Ancestor Table Field. The system automatically generates output fields based on the field names of the ancestor table. Manage Output Fields. You can also perform the following operations on the added fields: Click the Actions column's icon to edit existing fields. Click the Actions column's icon to delete the existing field.
   Mapping	Maps input fields from the source table to output fields in the destination table for data synchronization. Two mapping modes are available: Same-name Mapping: Maps fields with the same field name. Same-row Mapping: Maps fields based on their row position, regardless of field name differences between the source and destination tables.

9. Click Confirm to finalize the HDFS output widget configuration.