All Products
Search
Document Center

Export full data to MaxCompute

Last Updated: Mar 01, 2021

This topic describes how to export full data from HBase to MaxCompute.

Prerequisites

  1. Lindorm Tunnel Service (LTS) is purchased, the username and the password for logging on to the LTS web UI are set, and the LTS web UI is logged on.

  2. An HBase data source is added.

  3. A MaxCompute data source is added.

Features

1. Export full data from HBase to MaxCompute.

2. Specify the rowkey, the time range, and the columns.

Supported versions

  1. User-created HBase V1.x and V2.x

  2. E-MapReduce (EMR) HBase

  3. ApsaraDB for HBase Standard Edition and Performance-enhanced Edition that run in cluster mode and Lindorm

Create a task

  1. Log on to the LTS web UI. In the left-side navigation pane, choose Data Export > Archive Full Data to MaxCompute.

  2. On the Lindorm/HBase archive to MaxCompute page, click create new job.MaxCompute

  3. Select the source HBase cluster and the destination MaxCompute cluster, and specify the tables to be migrated.

    archive to MaxCompute
  4. Check the migration progress.

  5. After the migration is complete, log on to the MaxCompute console to query the data.

    View Result

Parameters

hbaseTable/odpsTable {"tableMode": "wideTable", "cols":["f1:a", "f1:b", "f1:c"]}
hbaseTable/odpsTable {"tableMode": "wideTable", "cols":["f1:a|string", "f1:b|boolean"]}
hbaseTable/odpsTable {"tableMode": "wideTable", "cols":["f1:a|string", "f1:b|boolean"], "startTs": "20190101000000", "endTs": "20200101000000", "startKey": "a", "endKey": "z"}
hbaseTable/odpsTable {"tableMode": "KVTable"}

You can use the key-value (KV) or wide table mode to export HBase data to MaxCompute.

KV mode

Input parameters: tablexxx {"tableMode": "KVTable"}

Response:

Result
Note
  • Each row indicates a KV pair in HBase.

  • rowkey: the rowkey of the KV pair in the HBase table.

  • version: the version of the KV pair.

  • op: the type of the KV pair. Valid values: Put, DeleteFamily, and DeleteColumn.

  • family: the column family.

  • qualifier: the name of the column.

  • value: the value of the KV pair. The value is converted to the HexString format.

Wide table mode

Input parameters: hbaseTable/odpsTable {"tableMode": "wideTable", "cols":["cf1:string|string", "cf1:int|int", "cf1:double|double"]}

Note

In wide table mode, you must specify the columns to be exported. For example, you can use "cols":["f1:a", "f1:b", "f1:c"]. By default, the column value is converted to the HexString format. You can specify the data type of the column. For example, you can use "cols":["f1:a|string", "f1:b|boolean", "f1:c|int"]. The columns support the following data types: string, int, long, short, decimal, double, float, and boolean. The default data type is hex. This indicates that byte arrays are converted to the HexString format.

Response:

Response

Specify the range of the data to be exported

Input parameters: table1 {"startKey": "xxx", "endKey": "zzz", "startTs": "20191001153000", "endTs": "20191001233000"}

When you export data from HBase to MaxCompute in KV mode or wide table mode, you can use the startKey and endKey parameters to specify the rowkey range. You can also use the startTs and endTs parameters to specify the time range.

Specify MaxCompute tables for exported data

You can export data to a MaxCompute table that has a different name than the source HBase table. By default, the MaxCompute table has the same name as the HBase table. The hyphens (-) and periods (.) in HBase table names are replaced with underscores (_). For example, you can use hbaseTable/odpsTable {"tableMode": "KVTable"}.