All Products
Search
Document Center

Import incremental data from Log Service

Last Updated: Feb 02, 2021

This topic describes how to import incremental data from Log Service.

Prerequisites

  • The Lindorm Tunnel Service (LTS) service that supports data migration and synchronization is purchased. The account and the password are set for the LTS web UI. The LTS web UI is logged on.

  • The LTS web UI is logged on. The HBase data source is created.

  • The LTS web UI is logged on. The LogHub data source is created.

Supported HBase versions

  • User-created HBase V1.x and V2.x clusters for which Kerberos is disabled.

  • E-MapReduce HBase.

  • ApsaraDB for HBase Standard Edition and Performance-enhanced Edition that run in cluster mode. ApsaraDB for HBase in single-node mode is not supported.

Create a task

  1. Log on to the LTS web UI. In the left-side navigation pane, choose Data Import > LogHub incremental Import.loghub-en-1

  2. Click Create Channel. On the page that appears, select the LogHub from which you want to import data in Source Data Source, and select an HBase data source in Destination Data Source (HBase, Phoenix). Then, configure the column mapping.loghub-en-1

  3. View the channel details after the channel is created.loghub-en-2

Table Mapping Parameter description

{
  "reader": {
    "columns": [
      "__client_ip__",
      "C_Source",
      "id",
      "name"
    ],
    "consumerSize": 2, // Specifies the number of consumers that subscribe to the LogHub data. The default value is 1.
    "logstore": "bds-test"
  },
  "writer": {
    "columns": [
      {
        "name": "f:col1",
        "value": "{{ concat('xx', name) }}" // The supported expression.
      },
      {
        "name": "f:col2",
        "value": "__client_ip__" // The mapped column name.
      },
      {
        "name": "f:*" // Columns that are not matched follow the default matching settings.
      }
    ],
    "rowkey": {
      "value": "{{ concat('idg',id) }}"
    },
    "tableName": "default:sls"
  }
}
  • The following simple expression is supported:

{
  "name": "cf1:hhh",
  "value": "{{ concat(title, id) }}"
}