Lindorm Tunnel Service (LTS) no longer supports incremental data import from Simple Log Service (SLS) as of March 10, 2023. If you purchased LTS after March 10, 2023, this feature is unavailable. If you purchased LTS before March 10, 2023, you can still use it.
Use LTS to stream incremental log data from SLS into HBase in real time — without writing custom consumers or managing offsets manually.
Prerequisites
Before you begin, ensure that you have:
-
Purchased LTS, configured your account credentials in the LTS console, and logged in
-
An ApsaraDB for HBase data source added in the LTS console
-
A loghub (SLS) data source added in the LTS console
Supported editions
| Edition | Supported |
|---|---|
| Self-managed HBase V1.x and V2.x (Kerberos disabled) | Yes |
| Alibaba Cloud E-MapReduce (EMR) HBase | Yes |
| ApsaraDB for HBase Standard Edition — cluster mode | Yes |
| ApsaraDB for HBase Performance-enhanced Edition — cluster mode | Yes |
| ApsaraDB for HBase — single-node mode | No |
Create a task
-
In the left-side navigation pane of the LTS console, select Data Import > SLS incremental Import.
-
Click create new job. From the Source Cluster drop-down list, select SLS. From the Target Cluster(HBase, Phoenix) drop-down list, select HBase.
-
Configure the table mapping field. See Table mapping parameters for details.
-
Click create.
After the job is created, view its details on the SLS incremental Import page.
Table mapping parameters
The table mapping is a JSON configuration with two top-level objects: reader (SLS source) and writer (HBase target).
Reader parameters
| Parameter | Required | Default | Description |
|---|---|---|---|
logstore |
Yes | — | Name of the SLS Logstore to read from |
columns |
Yes | — | Array of SLS log field names to include (e.g., "__client_ip__", "id") |
consumerSize |
No | 1 |
Number of consumers that subscribe to the loghub (SLS) data |
Writer parameters
| Parameter | Required | Default | Description |
|---|---|---|---|
tableName |
Yes | — | HBase table name in the format namespace:table (e.g., "default:sls") |
rowkey.value |
Yes | — | Expression that generates the row key for each record |
columns |
Yes | — | Array of column mapping objects (see below) |
Column mapping object fields:
| Field | Required | Description |
|---|---|---|
name |
Yes | HBase column in the format columnFamily:qualifier (e.g., "f:col1"). Use "f:*" to map unmatched columns with default settings. |
value |
No | SLS field name to map, or an expression using {{ }} syntax (e.g., "{{ concat('xx', name) }}") |
Expression syntax
Use {{ expression }} to compute column values or row keys dynamically. The concat() function concatenates multiple fields or string literals.
Examples:
{{ concat('idg', id) }} // Prepends "idg" to the id field {{ concat('xx', name) }} // Prepends "xx" to the name field {{ concat(title, id) }} // Concatenates title and idComplete configuration example
The following configuration reads four fields from an SLS Logstore named bds-test and writes them to the HBase table default:sls. It uses two consumers, applies expressions for f:col1 and rowkey, maps f:col2 directly from a log field, and routes unmatched columns with the f:* wildcard.
{
"reader": {
"columns": [
"__client_ip__",
"C_Source",
"id",
"name"
],
"consumerSize": 2,
"logstore": "bds-test"
},
"writer": {
"columns": [
{
"name": "f:col1",
"value": "{{ concat('xx', name) }}"
},
{
"name": "f:col2",
"value": "__client_ip__"
},
{
"name": "f:*"
}
],
"rowkey": {
"value": "{{ concat('idg', id) }}"
},
"tableName": "default:sls"
}
}