Dataphin:Configure the Amazon S3 Input Component

Prerequisites

• An Amazon S3 data source is created. Create an Amazon S3 Data Source.

• Your account has read-through permission for the data source. If not, Request Data Source Permission.

Procedure

1. On the Dataphin home page, select Development > Data Integration from the top menu bar.

2. In the top menu bar, select a Project. In Dev-Prod mode, also select an environment.

3. In the left-side navigation pane, click Batch Pipeline, then click the target Offline Pipeline in the Batch Pipeline list to open its configuration page.

4. In the upper-right corner, click Component Library to open the Component Library.

5. In the Component Library panel, select Input, find the Amazon S3 component, and drag it to the canvas.

6. Click the icon on the Amazon S3 component to open the Amazon S3 Input Configuration dialog box.

7. In the Amazon S3 Input Configuration dialog box, configure the parameters.

   Parameter	Description
   Step Name	Auto-generated component name. Editable. Naming rules: Supports Chinese characters, letters, underscores (_), and digits only. Maximum 64 characters.
   Datasource	All Amazon S3 data sources in Dataphin. Click the icon to copy a data source name. For data sources without read-through permission, click Request next to the data source name. Request Data Source Permission. To add a new data source, click Create Data Source. Create an Amazon S3 Data Source.
   Object Prefix	The key prefix of S3 objects within a bucket. Add multiple prefixes by clicking Click + Add Object Prefix. If a directory is configured in the data source, it auto-populates here. You can modify it, but ensure you have permission for the target directory or the task fails.
   File Type	Supported types: Text, CSV, xls, and xlsx. Parameters vary by type. For configuration parameters required for Text and CSV file types, see Text and CSV File Types. For configuration parameters required for xls and xlsx file types, see xls and xlsx File Types.
   File Encoding	Supported encodings: UTF-8 and GBK.
   Null Value Conversion	Empty by default. Specify a string to convert to NULL.
   Compression Format	Supported formats: zip, gzip, bzip2, lzo, and lzo_deflate.
   Output Fields	Output fields. Add fields manually: Click Batch Add. Add fields in JSON format. For example, "index":3,"name":"user_id","type":"String" imports the fourth column as a String field named user_id. Add fields in TEXT format. Separates fields. Default: line feed (\n). Also supports semicolon (;) and period (.). Separates the field name and type. Default: comma (,). Click Create Output Field and specify the Source Ordinal Number, Column, and Type. For Text and CSV files, the source ordinal number must be the zero-based column index. Manage added fields: Drag the icon to reorder fields. Click the icon in the Actions column to edit a field. Click the icon in the Actions column to delete a field.

   Text and CSV File Types

   Parameter	Description
   Column Delimiter	The delimiter between columns. Default: comma (,).
   Row Delimiter	The delimiter between rows. Default: line feed (\n).
   First Row Content Type	Select whether the first row contains field names or data content.

   xls and xlsx File Types

   Parameter	Description
   Sheet Selection	Select sheets by name or index. Multiple sheets must share the same data format.
   Sheet Name	Separate multiple sheet names with a comma (,), or enter * to read all sheets. Important * and comma (,) cannot be used together.
   Data Content Start Row	Default: 1 (first row). To skip the first N rows, set this to N+1.
   Data Content End Row	Optional. Default: the last row with data. Important The end row must be greater than or equal to the start row. Otherwise, the task reports an error.
   Export Sheet Name	Disabled by default. When enabled, adds a source sheet field to the output.

8. Click Confirm.