Overview of Lindorm Streams for real-time data subscription from HBase to Kafka - ApsaraDB for HBase - Alibaba Cloud - ApsaraDB for HBase

Lindorm Streams, provided by ApsaraDB for Lindorm, publishes incremental data from ApsaraDB for HBase Performance-enhanced to topics of Message Queue for Apache Kafka. Subscribe to these topics to consume row-level changes in real time for your downstream systems.

Lindorm Streams supports:

Ordered delivery per row
At-least-once delivery semantics
Before/after row state in each message
Configurable message parameters

How it works

Lindorm Streams publishes each row operation as a message to a Kafka topic. Each message can carry two versions of the affected row: the state before the operation and the state after the operation.

Delivery semantics

Order guarantee

Operations on the same row are published in the order they occur, so consumers can replay them in the same sequence.

At-least-once delivery

Lindorm Streams guarantees at-least-once delivery. Messages may be delivered more than once based on your retry policy, so design your consumer to handle duplicate messages.

To use exactly-once semantics, submit a ticket.

Message format

Each message is a JSON object with the following structure:

{
  "op": "<opType>",
  "table": "<tableName>",
  "ts": "<defaultVersion>",
  "keyOnly": "<keyOnly>",
  "data": [
    {
      "type": "<type>",
      "name": "<name>",
      "ts": "<version>",
      "value": "<value>"
    }
  ]
}

The following table describes the top-level fields.

Field	Description
`op`	The operation type. Valid values: `Put` and `DeleteFamily`.
`table`	The name of the table.
`ts`	The default version. Used when the row does not contain the `ts` key.
`keyOnly`	Whether only the primary key is included in the message.
`data`	An array of column entries. Each entry contains the fields described below.

The following table describes the fields in each data entry.

Field	Description
`type`	The data type of the column.
`name`	The name of the column.
`ts`	The data version. If blank, the `dts` value is used as the version.
`value`	The value of the column. If blank, the actual value in the table is `null`.

Message parameters

Use the following parameters to control which data is included in each message.

Parameter	Type	Required	Default	Description
`key_only`	boolean	No	false	Whether to synchronize only primary keys.
`unique_key`	boolean	No	false	Whether a primary key is sent only once per batch.