DataWorks:OSS-HDFS data source

Limits

Batch data read

• When you use OSS-HDFS Reader, take note of the following items:

  Network connections between a resource group and OSS-HDFS are complex. We recommend that you use an exclusive resource group for Data Integration to run your data synchronization task to which the OSS-HDFS data source is added. Make sure that your exclusive resource group for Data Integration can access the network where the OSS-HDFS data source resides.

• OSS-HDFS Reader supports the following features:

  • Supports the text, CSV, ORC, and Parquet file formats. Data stored in the files in these formats must be organized as logical two-dimensional tables.

  • Reads data of various types and supports constants.

  • Supports recursive reading and wildcards asterisks (*) and question marks (?).

  • Uses parallel threads to read data from multiple files.

Batch data write

Take note of the following items when you use OSS-HDFS Writer:

• OSS-HDFS Writer can write only text, ORC, and Parquet files that store logical two-dimensional tables to OSS-HDFS.

• To write a text file to OSS-HDFS, make sure that the delimiter in the file is the same as that in the Hive table that you want to associate with the file. This way, you can associate the columns in the file that is written to OSS-HDFS with those in the Hive table.

How it works

Data type mappings

Batch data read

The following table lists the data type mappings based on which OSS-HDFS Reader converts data types in Parquet, ORC, text, and CSV files.

Batch data write

OSS-HDFS Writer can write text, ORC, or Parquet files to a specified directory in OSS-HDFS.

Develop a data synchronization task

For information about the entry point for and the procedure of configuring a data synchronization task, see the following sections. For information about the parameter settings, view the infotip of each parameter on the configuration tab of the task.

Add a data source

Before you configure a data synchronization task to synchronize data from or to a specific data source, you must add the data source to DataWorks. For more information, see Add and manage data sources.

Configure a batch synchronization task to synchronize data of a single table

• For more information about the configuration procedure, see Configure a batch synchronization task by using the codeless UI and Configure a batch synchronization task by using the code editor.

• For information about all parameters that are configured and the code that is run when you use the code editor to configure a batch synchronization task, see Appendix: Code and parameters.

Appendix: Code and parameters

Appendix: Configure a batch synchronization task by using the code editor

If you use the code editor to configure a batch synchronization task, you must configure parameters for the reader and writer of the related data source based on the format requirements in the code editor. For more information about the format requirements, see Configure a batch synchronization task by using the code editor. The following information describes the configuration details of parameters for the reader and writer in the code editor.

Code for OSS-HDFS Reader

{
    "type": "job",
    "version": "2.0",
    "steps": [
        {
            "stepType": "oss_hdfs",// The plug-in name.
            "parameter": {
                "path": "",// The path of the file from which you want to read data.
                "datasource": "",// The name of the data source.
                "column": [
                    {
                        "index": 0,// The index of the column in the source file. The index starts from 0, which indicates that OSS-HDFS Reader reads data from the first column of the source file. 
                        "type": "string"// The field type.
                    },
                    {
                        "index": 1,
                        "type": "long"
                    },
                    {
                        "index": 2,
                        "type": "double"
                    },
                    {
                        "index": 3,
                        "type": "boolean"
                    },
                    {
                        "format": "yyyy-MM-dd HH:mm:ss", // The time format.
                        "index": 4,
                        "type": "date"
                    }
                ],
                "fieldDelimiter": ",",// The column delimiter.
                "encoding": "UTF-8",// The encoding format.
                "fileFormat": ""// The file format.
            },
            "name": "Reader",
            "category": "reader"
        },
        {
            "stepType": "stream",
            "parameter": {},
            "name": "Writer",
            "category": "writer"
        }
    ],
    "setting": {
        "errorLimit": {
            "record": ""// The maximum number of dirty data records allowed.
        },
        "speed": {
            "concurrent": 3,// The maximum number of parallel threads.
            "throttle": true // Specifies whether to enable throttling. The value false indicates that throttling is disabled, and the value true indicates that throttling is enabled. The mbps parameter takes effect only when the throttle parameter is set to true. 
            "mbps":"12"// The maximum transmission rate. Unit: MB/s. 
        }
    },
    "order": {
        "hops": [
            {
                "from": "Reader",
                "to": "Writer"
            }
        ]
    }
}

Parameters in code for OSS-HDFS Reader

{
  "type": "long",
  "index": 0
  // The first LONG-type column of the source file. The index starts from 0. The index field indicates the IDs of the columns from which you want to read data in the file. 
},
{
  "type": "string",
  "value": "alibaba"
  // The value of the current column, which is a constant column alibaba. It is internally generated by OSS-HDFS Reader. 
}

Code for OSS-HDFS Writer

{
    "type": "job",
    "version": "2.0",// The version number. 
    "steps": [
        {
            "stepType": "stream",
            "parameter": {},
            "name": "Reader",
            "category": "reader"
        },
        {
            "stepType": "oss_hdfs",// The plug-in name. 
            "parameter": {
                "path": "",// The directory in OSS-HDFS to which the files are written. 
                "fileName": "",// The name prefix of the files that you want to write to OSS-HDFS. 
                "compress": "",// The compression format of the files that you want to write to OSS-HDFS. 
                "datasource": "",// The name of the data source. 
                "column": [
                    {
                        "name": "col1",// The name of a column. 
                        "type": "string"// The data type of a column. 
                    },
                    {
                        "name": "col2",
                        "type": "int"
                    },
                    {
                        "name": "col3",
                        "type": "double"
                    },
                    {
                        "name": "col4",
                        "type": "boolean"
                    },
                    {
                        "name": "col5",
                        "type": "date"
                    }
                ],
                "writeMode": "",// The write mode. 
                "fieldDelimiter": ",",// The column delimiter. 
                "encoding": "",// The encoding format. 
                "fileFormat": "text"// The format of the files that you want to write to OSS-HDFS. 
            },
            "name": "Writer",
            "category": "writer"
        }
    ],
    "setting": {
        "errorLimit": {
            "record": ""// The maximum number of dirty data records allowed. 
        },
        "speed": {
            "concurrent": 3,// The maximum number of parallel threads. 
            "throttle": false // Specifies whether to enable throttling. The value false indicates that throttling is disabled, and the value true indicates that throttling is enabled. The mbps parameter takes effect only when the throttle parameter is set to true. 
        }
    },
    "order": {
        "hops": [
            {
                "from": "Reader",
                "to": "Writer"
            }
        ]
    }
}

Parameters in code for OSS-HDFS Writer

{
    "column":
    [
        {
            "name": "userName",
            "type": "string"
        },
        {
            "name": "age",
            "type": "long"
        }
    ]
}

message MessageName {
required, dataType, columnName;
.....................;
}

message m {
optional int64 id;
optional int64 date_id;
optional binary datetimestring;
optional int32 dspId;
optional int32 advertiserId;
optional int32 status;
optional int64 bidding_req_num;
optional int64 imp;
optional int64 click_num;
}