HDFS Writer

Limits

• Alibaba Cloud Apsara File Storage for HDFS is not supported.
• HDFS Writer can write only text, ORC, and Parquet files that store logical two-dimensional tables to HDFS.
• HDFS is a distributed file system and does not have a schema. Therefore, you cannot write only data in some columns of a file to HDFS.
• Hive data types, such as DECIMAL, BINARY, ARRAYS, MAPS, STRUCTS, and UNION, are not supported.
• HDFS Writer can write data to only one partition in a partitioned Hive table at a time.
• To write a text file to HDFS, make sure that the delimiter in the file is the same as that in the Hive table that you want to associate with the file. This way, you can associate the columns in the file that is written to HDFS with those in the Hive table.
• You can use HDFS Writer in the environment in which Hive 1.1.1 and Hadoop 2.7.1 (JDK version: 1.7) are installed. JDK is short for Java Development Kit. HDFS Writer can write files to HDFS in test environments in which Hive 1.2.0 and Hadoop 2.5.0 or Hadoop 2.6.0 are installed.

How it works

Data types

HDFS Writer supports most Hive data types. Make sure that the data types of your system are supported.

The following table lists the Hive data types that are supported by HDFS Writer.

Parameters

"column": 
[
    {
        "name": "userName",
        "type": "string"
    },
    {
        "name": "age",
        "type": "long"
    }
]

message MessageTypeName {
required, dataType, columnName;
......................;
}

message m {
optional int64 id;
optional int64 date_id;
optional binary datetimestring;
optional int32 dspId;
optional int32 advertiserId;
optional int32 status;
optional int64 bidding_req_num;
optional int64 imp;
optional int64 click_num;
}

"hadoopConfig":{
"dfs.nameservices": "testDfs",
"dfs.ha.namenodes.testDfs": "namenode1,namenode2",
"dfs.namenode.rpc-address.youkuDfs.namenode1": "",
"dfs.namenode.rpc-address.youkuDfs.namenode2": "",
"dfs.client.failover.proxy.provider.testDfs": "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider"
}

```json
    "writer": {
    "name": "hdfswriter",
    "parameter": {
        "defaultFS": "oss://test-bucket",
        "fileType": "parquet",
        "path": "/datasets/oss_demo/kpt",
        "fileName": "test",
        "writeMode": "truncate",
        "compress": "SNAPPY",
        "encoding": "UTF-8",
        "hadoopConfig": {
            "fs.oss.accessKeyId": "the-access-id",
            "fs.oss.accessKeySecret": "the-access-key",
            "fs.oss.endpoint": "oss-cn-hangzhou.aliyuncs.com"
            },
            "parquetSchema": "message test {\n    required int64 id;\n    optional binary name (UTF8);\n    optional int64 gmt_create;\n    required group map_col (MAP) {\n        repeated group key_value {\n            required binary key (UTF8);\n            required binary value (UTF8);\n        }\n    }\n    required group array_col (LIST) {\n        repeated group list {\n            required binary element (UTF8);\n        }\n    }\n    required group struct_col {\n        required int64 id;\n        required binary name (UTF8);\n    }    \n}",
            "dataxParquetMode": "fields"
            }
        }
    ```

"haveKerberos":true,
"kerberosKeytabFilePath":"/opt/datax/**.keytab",
"kerberosPrincipal":"**/hadoopclient@**.**"

Configure HDFS Writer by using the codeless UI

1. Configure data sources.
   Configure Source and Target for the synchronization node.

   Parameter	Description
   Connection	The name of the data source to which you want to write data. This parameter is equivalent to the datasource parameter that is described in the preceding section.
   File path	The directory in HDFS to which you want to write files. This parameter is equivalent to the path parameter that is described in the preceding section.
   File type	The format of the files that you want to write to HDFS. This parameter is equivalent to the fileType parameter that is described in the preceding section. Valid values: text, orc, and parquet.
   File name	The name prefix of the files that you want to write to HDFS. This parameter is equivalent to the fileName parameter that is described in the preceding section.
   WriteMode	The write mode. This parameter is equivalent to the writeMode parameter that is described in the preceding section. Valid values: append: HDFS Writer writes the files based on the specified file name prefix and ensures that the actual file names do not conflict with the names of existing files. nonConflict: HDFS Writer returns an error if a file with the specified file name prefix exists in the destination directory. Note Parquet files do not support the append mode. They support only the nonConflict mode.
   FieldDelimiter	The column delimiter that is used in the files you want to write to HDFS. This parameter is equivalent to the fieldDelimiter parameter that is described in the preceding section. Make sure that you use the same delimiter as that in the Hive table. Otherwise, you cannot query data in the Hive table.
   Encoding	The encoding format. This parameter is equivalent to the encoding parameter that is described in the preceding section. Default value: UTF-8.
   Kerberos authentication	Specifies whether Kerberos authentication is required. Default value: No. If you set this parameter to Yes, you must also specify the KeyTab file path and Principal Name parameters. For more information, see Configure Kerberos authentication.
   HadoopConfig	The settings of the advanced Hadoop parameters that are related to high availability. If you use the shared resource group for Data Integration, you cannot configure the advanced Hadoop parameters that are related to high availability.

2. Configure field mappings. This operation is equivalent to setting the column parameter that is described in the preceding section. By default, Map Fields in the Same Line is used to establish mappings between fields. You can click the icon to edit fields. Each field occupies a row. The first and the last blank rows are included, whereas other blank rows are ignored. Make sure that the numbers of fields in the source and destination tables match.

Configure HDFS Writer by using the code editor

For more information about how to configure a synchronization node by using the code editor, see Create a synchronization node by using the code editor.

{
    "type": "job",
    "version": "2.0",// The version number. 
    "steps": [
        { 
            "stepType": "stream",
            "parameter": {},
            "name": "Reader",
            "category": "reader"
        },
        {
            "stepType": "hdfs",// The writer type. 
            "parameter": {
                "path": "",// The directory in HDFS to which the files are written. 
                "fileName": "",// The name prefix of the files that you want to write to HDFS. 
                "compress": "",// The compression format of the files that you want to write to HDFS. 
                "datasource": "",// The name of the data source. 
                "column": [
                    {
                        "name": "col1",// The name of a column. 
                        "type": "string"// The data type of a column. 
                    },
                    {
                        "name": "col2",
                        "type": "int"
                    },
                    {
                        "name": "col3",
                        "type": "double"
                    },
                    {
                        "name": "col4",
                        "type": "boolean"
                    },
                    {
                        "name": "col5",
                        "type": "date"
                    }
                ],
                "writeMode": "",// The write mode. 
                "fieldDelimiter": ",",// The column delimiter. 
                "encoding": "",// The encoding format. 
                "fileType": "text"// The format of the files that you want to write to HDFS. 
            },
            "name": "Writer",
            "category": "writer"
        }
    ],
    "setting": {
        "errorLimit": {
            "record": ""// The maximum number of dirty data records allowed. 
        },
        "speed": {
            "throttle":true,// Specifies whether to enable bandwidth throttling. The value false indicates that bandwidth throttling is disabled, and the value true indicates that bandwidth throttling is enabled. The mbps parameter takes effect only when the throttle parameter is set to true. 
            "concurrent":3, // The maximum number of parallel threads. 
            "mbps":"12"// The maximum transmission rate.
        }
    },
    "order": {
        "hops": [
            {
                "from": "Reader",
                "to": "Writer"
            }
        ]
    }
}