Lindorm:Change tracking

Background information

The change tracking feature of Lindorm records every data change in a table. You can view these change records on your client. After you enable the change tracking feature for a table, each time you insert, update, or delete data, a stream record is generated. This record is a key-value pair. The key is the primary key of the row. The value includes details about the operation, such as the value before and after the operation, the timestamp, and the operation type.

Enablement modes

You can enable the Lindorm change tracking feature in two modes: pull model and push model.

• Pull model: You can use an open source Kafka client to subscribe to data directly. The service temporarily stores the data. You must set an expiration time for the stored data, after which it is deleted. For more information, see Create a data subscription channel in pull model.

• Push model: The Lindorm change tracking feature pushes data directly to a Kafka client. The service does not store the data. For more information, see Create a data subscription in push model.

Data subscription lifecycle

• If you release the Lindorm Tunnel Service (LTS) cluster without first stopping the task, the sync task pauses. This prevents the data from being consumed.

• If you enable change tracking but do not consume the data, the subscription relationship is automatically canceled by default after 48 hours, and the data is also deleted.

Limits

• The change tracking feature of Lindorm supports only at-least-once processing semantics. This means the same data record may be consumed multiple times due to internal retries.

• Incremental messages from the Lindorm change tracking feature are out of order. The order in which you consume messages is not the same as the order in which they are written to the Lindorm wide table.

• The change tracking feature for HBase tables requires Lindorm Tunnel Service (LTS) version 3.7.0 or later.

  Note

  • If you have enabled LTS, upgrade it to the latest version before you enable the change tracking feature. For more information, see Minor version update.

  • If you have not enabled LTS, you can purchase it when you enable the change tracking feature. The latest version of LTS is purchased by default. For more information, see Enable change tracking.

• We recommend that you do not enable the change tracking feature for non-SSD instances. This is because when this feature is used to export the latest or pre-update value of a row, an additional read operation is performed to retrieve the pre-update value. As a result, each write operation incurs an additional read operation. If you do not need the OldImage or NewImage feature and only require the delta data, please submit a ticket.

• When you create a data subscription channel in the pull model, the subscribed data is stored for seven days by default. This data shares the underlying Lindorm Distributed File System (LDFS) storage space with Lindorm. Before you enable the change tracking feature, make sure that you have sufficient storage space.