DataWorks:HBase data source

HBase Reader

HBase Reader supports normal and multiVersionFixedColumn modes.

• normal mode: Reads an HBase table as a standard two-dimensional table and retrieves the latest version of the data.

  hbase(main):017:0> scan 'users'
  ROW                                   COLUMN+CELL
  lisi                                 column=address:city, timestamp=1457101972764, value=beijing
  lisi                                 column=address:contry, timestamp=1457102773908, value=china
  lisi                                 column=address:province, timestamp=1457101972736, value=beijing
  lisi                                 column=info:age, timestamp=1457101972548, value=27
  lisi                                 column=info:birthday, timestamp=1457101972604, value=1987-06-17
  lisi                                 column=info:company, timestamp=1457101972653, value=baidu
  xiaoming                             column=address:city, timestamp=1457082196082, value=hangzhou
  xiaoming                             column=address:contry, timestamp=1457082195729, value=china
  xiaoming                             column=address:province, timestamp=1457082195773, value=zhejiang
  xiaoming                             column=info:age, timestamp=1457082218735, value=29
  xiaoming                             column=info:birthday, timestamp=1457082186830, value=1987-06-17
  xiaoming                             column=info:company, timestamp=1457082189826, value=alibaba
  2 row(s) in 0.0580 seconds }

  The following table shows the data after it is read.

  rowKey	address:city	address:contry	address:province	info:age	info:birthday	info:company
  lisi	beijing	china	beijing	27	1987-06-17	baidu
  xiaoming	hangzhou	china	zhejiang	29	1987-06-17	alibaba

• multiVersionFixedColumn mode: Reads an HBase table as a vertical table. Each record that is read has a four-column format: rowKey, family:qualifier, timestamp, and value. You must specify the columns to read. The value in each cell is treated as a record. If multiple versions exist, multiple records are generated.

  hbase(main):018:0> scan 'users',{VERSIONS=>5}
  ROW                                   COLUMN+CELL
  lisi                                 column=address:city, timestamp=1457101972764, value=beijing
  lisi                                 column=address:contry, timestamp=1457102773908, value=china
  lisi                                 column=address:province, timestamp=1457101972736, value=beijing
  lisi                                 column=info:age, timestamp=1457101972548, value=27
  lisi                                 column=info:birthday, timestamp=1457101972604, value=1987-06-17
  lisi                                 column=info:company, timestamp=1457101972653, value=baidu
  xiaoming                             column=address:city, timestamp=1457082196082, value=hangzhou
  xiaoming                             column=address:contry, timestamp=1457082195729, value=china
  xiaoming                             column=address:province, timestamp=1457082195773, value=zhejiang
  xiaoming                             column=info:age, timestamp=1457082218735, value=29
  xiaoming                             column=info:age, timestamp=1457082178630, value=24
  xiaoming                             column=info:birthday, timestamp=1457082186830, value=1987-06-17
  xiaoming                             column=info:company, timestamp=1457082189826, value=alibaba
  2 row(s) in 0.0260 seconds }

  The following table shows the data (four columns) after it is read.

  rowKey	column:qualifier	timestamp	value
  lisi	address:city	1457101972764	beijing
  lisi	address:contry	1457102773908	china
  lisi	address:province	1457101972736	beijing
  lisi	info:age	1457101972548	27
  lisi	info:birthday	1457101972604	1987-06-17
  lisi	info:company	1457101972653	beijing
  xiaoming	address:city	1457082196082	hangzhou
  xiaoming	address:contry	1457082195729	china
  xiaoming	address:province	1457082195773	zhejiang
  xiaoming	info:age	1457082218735	29
  xiaoming	info:age	1457082178630	24
  xiaoming	info:birthday	1457082186830	1987-06-17
  xiaoming	info:company	1457082189826	alibaba

HBase Writer

• rowkey generation rule: The HBase Writer supports concatenating multiple source fields to generate the rowkey for the HBase table.

• Version (timestamp) support for writing data to HBase:

  • Use the current time as the version.

  • Use a source column as the version.

  • Use a specific time as the version.

Offline read

• The following table lists the readable HBase data types and the mappings between HBase types and HBase Reader types.

  Category	Data Integration column type	Database data type
  Integer	long	short, int, and long
  Floating-point	double	float and double
  String	string	binary_string and string
  Date and time	date	date
  Byte	bytes	bytes
  Boolean	boolean	boolean

• HBase20xsql Reader supports most Phoenix types. However, some types are not supported. You must check your data types.

• The following table lists the mappings between Phoenix types and HBase20xsql Reader types.

  DataX internal type	Phoenix data type
  long	INTEGER, TINYINT, SMALLINT, BIGINT
  double	FLOAT, DECIMAL, DOUBLE
  string	CHAR, VARCHAR
  date	DATE, TIME, TIMESTAMP
  bytes	BINARY, VARBINARY
  boolean	BOOLEAN

Offline write

The following table lists the writable HBase data types and the mappings between HBase types and HBase Writer types.

Configuration guide for single-table offline synchronization tasks

• For more information, see Configure a sync task in the codeless UI and Configure a sync task in the code editor.

  When you use the codeless UI, field mappings are not displayed by default because HBase is a data source with no fixed structure. You must manually configure the field mappings:

  • If HBase is the source, configure the Source Field. The format is: Field Type|Column Family:Column Name.

  • If HBase is the destination, configure the Destination Field and rowkey. The format for Destination Field is Source Field Index|Field Type|Column Family:Column Name. The format for rowkey is Source Primary Key Index|Field Type.

  Note

  Enter each field on a new line.

• For a complete list of parameters and a script demo for the code editor, see Appendix: Script demos and parameter descriptions.

Configure a batch synchronization task by using the code editor

If you want to configure a batch synchronization task by using the code editor, you must configure the related parameters in the script based on the unified script format requirements. For more information, see Configure a task in the code editor. The following information describes the parameters that you must configure for data sources when you configure a batch synchronization task by using the code editor.

HBase Reader script demo

{
    "type":"job",
    "version":"2.0",// The version number.
    "steps":[
        {
            "stepType":"hbase",// The plugin name.
            "parameter":{
                "mode":"normal",// The mode to read data from HBase. Valid values: normal and multiVersionFixedColumn.
                "scanCacheSize":"256",// The number of rows that the HBase client reads from the server in each remote procedure call (RPC).
                "scanBatchSize":"100",// The number of columns that the HBase client reads from the server in each RPC. 
                "hbaseVersion":"094x/11x",// The HBase version.
                "column":[// The fields.
                    {
                        "name":"rowkey",// The field name.
                        "type":"string"// The data type.
                    },
                    {
                        "name":"columnFamilyName1:columnName1",
                        "type":"string"
                    },
                    {
                        "name":"columnFamilyName2:columnName2",
                        "format":"yyyy-MM-dd",
                        "type":"date"
                    },
                    {
                        "name":"columnFamilyName3:columnName3",
                        "type":"long"
                    }
                ],
                "range":{// Specifies the range of rowkeys for HBase Reader to read.
                    "endRowkey":"",// The end rowkey.
                    "isBinaryRowkey":true,// Specifies how to convert the configured startRowkey and endRowkey to byte arrays. The default value is false.
                    "startRowkey":""// The start rowkey.
                },
                "maxVersion":"",// The number of versions that HBase Reader reads in multi-version mode.
                "encoding":"UTF-8",// The encoding format.
                "table":"",// The table name.
                "hbaseConfig":{// The configuration information required to connect to the HBase cluster, in JSON format.
                    "hbase.zookeeper.quorum":"hostname",
                    "hbase.rootdir":"hdfs://ip:port/database",
                    "hbase.cluster.distributed":"true"
                }
            },
            "name":"Reader",
            "category":"reader"
        },
        {
            "stepType":"stream",
            "parameter":{},
            "name":"Writer",
            "category":"writer"
        }
    ],
    "setting":{
        "errorLimit":{
            "record":"0"// The number of error records.
        },
        "speed":{
            "throttle":true,// If throttle is set to false, the mbps parameter does not take effect and no rate limiting is applied. If throttle is set to true, rate limiting is applied.
            "concurrent":1,// The number of concurrent jobs.
            "mbps":"12"// The rate limit. 1 mbps = 1 MB/s.
        }
    },
    "order":{
        "hops":[
            {
                "from":"Reader",
                "to":"Writer"
            }
        ]
    }
}

HBase Reader script parameters

HBase Writer script demo

{
    "type":"job",
    "version":"2.0",// The version number.
    "steps":[
        {
            "stepType":"stream",
            "parameter":{},
            "name":"Reader",
            "category":"reader"
        },
        {
            "stepType":"hbase",// The plugin name.
            "parameter":{
                "mode":"normal",// The mode to write data to HBase.
                "walFlag":"false",// Disables (false) writing WAL logs.
                "hbaseVersion":"094x",// The HBase version.
                "rowkeyColumn":[// The rowkey column to write to HBase.
                    {
                        "index":"0",// The serial number.
                        "type":"string"// The data type.
                    },
                    {
                        "index":"-1",
                        "type":"string",
                        "value":"_"
                    }
                ],
                "nullMode":"skip",// How to handle null values.
                "column":[// The HBase fields to write to.
                    {
                        "name":"columnFamilyName1:columnName1",// The field name.
                        "index":"0",// The index number.
                        "type":"string"// The data type.
                    },
                    {
                        "name":"columnFamilyName2:columnName2",
                        "index":"1",
                        "type":"string"
                    },
                    {
                        "name":"columnFamilyName3:columnName3",
                        "index":"2",
                        "type":"string"
                    }
                ],
                "encoding":"utf-8",// The encoding format.
                "table":"",// The table name.
                "hbaseConfig":{// The configuration information required to connect to the HBase cluster, in JSON format.
                    "hbase.zookeeper.quorum":"hostname",
                    "hbase.rootdir":"hdfs: //ip:port/database",
                    "hbase.cluster.distributed":"true"
                }
            },
            "name":"Writer",
            "category":"writer"
        }
    ],
    "setting":{
        "errorLimit":{
            "record":"0"// The number of error records.
        },
        "speed":{
            "throttle":true,// If throttle is set to false, the mbps parameter does not take effect and no rate limiting is applied. If throttle is set to true, rate limiting is applied.
            "concurrent":1, // The number of concurrent jobs.
            "mbps":"12"// The rate limit.
        }
    },
    "order":{
        "hops":[
            {
                "from":"Reader",
                "to":"Writer"
            }
        ]
    }
}

HBase Writer script parameters

"rowkeyColumn": [
          {
            "index":0,
            "type":"string"
          },
          {
            "index":-1,
            "type":"string",
            "value":"_"
          }
      ]

HBase20xsql Reader script demo

{
    "type":"job",
    "version":"2.0",// The version number.
    "steps":[
        {
            "stepType":"hbase20xsql",// The plugin name.
            "parameter":{
                "queryServerAddress": "http://127.0.0.1:8765",  // The Phoenix QueryServer endpoint.
                "serialization": "PROTOBUF",  // The QueryServer serialization format.
                "table": "TEST",    // The table to read.
                "column": ["ID", "NAME"],   // The columns to read.
                "splitKey": "ID"    // The sharding column, which must be the primary key of the table.
            },
            "name":"Reader",
            "category":"reader"
        },
        {
            "stepType":"stream",
            "parameter":{},
            "name":"Writer",
            "category":"writer"
        }
    ],
    "setting":{
        "errorLimit":{
            "record":"0"// The number of error records.
        },
        "speed":{
            "throttle":true,// If throttle is set to false, the mbps parameter does not take effect and no rate limiting is applied. If throttle is set to true, rate limiting is applied.
            "concurrent":1,// The number of concurrent jobs.
            "mbps":"12"// The rate limit. 1 mbps = 1 MB/s.
        }
    },
    "order":{
        "hops":[
            {
                "from":"Reader",
                "to":"Writer"
            }
        ]
    }
}

HBase20xsql Reader script parameters

HBase11xsql Writer script demo

{
  "type": "job",
  "version": "1.0",
  "configuration": {
    "setting": {
      "errorLimit": {
        "record": "0"
      },
      "speed": {
            "throttle":true,// If throttle is set to false, the mbps parameter does not take effect and no rate limiting is applied. If throttle is set to true, rate limiting is applied.
            "concurrent":1, // The number of concurrent jobs.
            "mbps":"1"// The rate limit. 1 mbps = 1 MB/s.
      }
    },
    "reader": {
      "plugin": "odps",
      "parameter": {
        "datasource": "",
        "table": "",
        "column": [],
        "partition": ""
      }
    },
    "plugin": "hbase11xsql",
    "parameter": {
      "table": "The destination HBase table name, case-sensitive",
      "hbaseConfig": {
        "hbase.zookeeper.quorum": "The ZooKeeper server endpoint of the destination HBase cluster",
        "zookeeper.znode.parent": "The znode of the destination HBase cluster"
      },
      "column": [
        "columnName"
      ],
      "batchSize": 256,
      "nullMode": "skip"
    }
  }
}

HBase11xsql Writer script parameters