Lindorm provides the change tracking feature to help you obtain incremental data in Lindorm database tables in real time. You can consume the obtained incremental data to develop or implement your business.
Background information
The change tracking feature records the change of each row of data in a Lindorm database table. You can view the change records on your client in real time. After you enable the change tracking feature for a Lindorm database table, each operation that modifies the data in the table is recorded. When you insert, update, or delete data, Lindorm generates a stream record that is a key-value pair. In the key-value pair, the key is the primary key value of the row in which the data resides, and the value includes the details of the operation. The details include the data before the operation is performed, the data after the operation is performed, the timestamp when the operation is performed, and the type of the operation.
Enable change tracking
You can use one of the following modes to enable change tracking: Pull and Push.
Pull mode: You can use an open source Kafka client to subscribe to data. The subscribed data is temporarily stored on the Lindorm server. You must specify a duration for which you want to retain the subscribed data. When the data is retained for a period longer than the specified duration, the data is deleted. For more information, see Create a Pull channel for data subscription.
Push mode: Lindorm pushes the subscribed data to your Message Queue for Apache Kafka topic. The Lindorm server does not store the subscribed data. For more information, see Create a Push channel for data subscription.
Limits
The change tracking feature supports only the At Least Once data processing semantics. Data can be consumed multiple times based on the retry policy that you configure in Lindorm Tunnel Service (LTS).
You can enable the change tracking feature only for tables that are created by executing Lindorm SQL statements. If you want to enable the change tracking feature for Lindorm HBase database tables, submit a ticket.
If your Lindorm instance does not use SSDs, we recommend that you do not enable the change tracking feature. If you enable this feature for a database table in a Lindorm instance that does not use SSDs, the database performance may be affected. After you enable the change tracking feature for a table, each time the latest data in a row or data in a row before update is exported, Lindorm reads the data in the row before the data is exported. A read operation is performed each time when you perform a write operation. If you want to subscribe to only the delta data and do not want to use the OldImage/NewImage feature,submit a ticket.
If you use the Pull mode to create data subscription channels, data is stored in LindormDFS for seven days by default. Make sure that your LindormDFS instance has sufficient storage space before you enable the change tracking feature.