All Products
Search
Document Center

Features

Last Updated: May 19, 2022

This topic describes how Lindorm Streams provided by ApsaraDB for Lindorm is used to publish messages to topics of Message Queue for Apache Kafka. The messages contain incremental data from ApsaraDB for HBase Performance-enhanced. This way, you can subscribe to the topics and consume the incremental data.

Features

Lindorm Streams publishes messages to topics of Message Queue for Apache Kafka based on the following rules:

  • Lindorm Streams records operations that are performed on the same row in the order in which these operations are performed. This way, the recorded operations can be re-performed in the same order.

  • The At-least-once semantics is supported. Messages can be consumed multiple times based on the retry policy that you configure. If you want to specify the exactly-once semantics, submit a ticket.

Message format

A Lindorm Streams message can contain two versions of an updated row: the version before the row is updated and the version after the row is updated. The following example shows the content of a message:

{
  "op" : <opType>             // The operation that is performed. Valid values: Put and DeleteFamily.
  "table" : <tableName>       // The name of the table.
  "ts" : <defaultVersion>     // The default version. The default version is used if the row does not contain the ts key.
  "keyOnly":<keyOnly>         // Indicates whether only the primary key is included.
  "data" : [
    {
      "type" : <type>          // The data type of the column.
      "name" : <name>          // The name of the column.
      "ts" : <version>         // The data version. If this parameter is empty, the dts value is used as the version.
      "value" : <value>        // The value of the column. If this parameter is empty, the actual value in the table is null.
  },
  {
      "type" : <type>          // The data type of the column.
      "name" : <name>          // The name of the column.
      "ts" : <version>         // The data version. If this parameter is empty, the dts value is used as the version.
      "value" : <value>        // The value of the column. If this parameter is empty, the actual value in the table is null.
  }
  ]
}

The following table describes the parameters that you can specify for Lindorm Streams messages.

Parameter

Type

Required

Example

Description

key_only

boolean

No

false

Specifies whether to synchronize only primary keys. Default value: false.

unique_key

boolean

No

false

Specifies whether a primary key is sent only once per batch. Default value: false.