You can use Function Compute to perform real-time computing on incremental data stored in Tablestore.
Background information
Tablestore Stream is a data channel used to obtain incremental data in Tablestore tables. By creating Tablestore triggers, Tablestore Stream and Function Compute can be automatically docked. This allows the custom program logic in the computing function to automatically process changed data in Tablestore tables.
Scenarios
The following figure shows the tasks that you can use Function Compute to perform.
- Data synchronization: You can use Function Compute to synchronize real-time data stored in Tablestore to data caches, search engines, or other database instances.
- Data archiving: You can use Function Compute to incrementally archive data stored in Tablestore to OSS for cold archiving.
- Event-driven application: You can create triggers to trigger functions to call API operations provided by IoT suites or cloud-based applications or send notifications.
Configure a Tablestore trigger
You can create a Tablestore trigger in the Tablestore console to process the real-time data stream generated by the incremental data in a Tablestore table.
Data Processing
- Data format
A Tablestore trigger encodes the incremental data in the CBOR format to construct a Function Compute events. The following code provides an example of the format of incremental data:
{ "Version": "string", "Records": [ { "Type": "string", "Info": { "Timestamp": int64 }, "PrimaryKey": [ { "ColumnName": "string", "Value": formated_value } ], "Columns": [ { "Type": "string", "ColumnName": "string", "Value": formated_value, "Timestamp": int64 } ] } ] }
- Elements
The following table describes the elements included in the preceding format.
Element Description Version The version of the payload, which is Sync-v1. Data type: string. Records The array that contains the incremental data in the table. This element includes the following members: - Type: the type of the columns. Valid values: PutRow, UpdateRow, and DeleteRow. Data type: string.
- Info: includes the Timestamp member, which indicates the last time when the row was modified. The value of Timestamp must be in the UTC format. Data type: int64.
PrimaryKey The array that stores the primary key column. This element includes the following members: - ColumnName: the name of the primary key column. Data type: string.
- Value: the content of the primary key column. Data type: formated_value. Valid values: integer, string, and blob.
Colunms The array that stores the attribute columns. This element includes the following members: - Type: the type of the attribute column. Valid values: PutOneVersion, DeleteOneVersion, and DeleteAllVersions. Data type: string.
- ColumnName: the name of the attribute column. Data type: string.
- Value: the content of the attribute column. Data type: formated_value. Valid values: integer, boolean, double, string, and blob.
- Timestamp: the last time when the attribute column was modified. The value of Timestamp must be in the UTC format. Data type: int64.
- Sample data
{ "Version": "Sync-v1", "Records": [ { "Type": "PutRow", "Info": { "Timestamp": 1506416585740836 }, "PrimaryKey": [ { "ColumnName": "pk_0", "Value": 1506416585881590900 }, { "ColumnName": "pk_1", "Value": "2017-09-26 17:03:05.8815909 +0800 CST" }, { "ColumnName": "pk_2", "Value": 1506416585741000 } ], "Columns": [ { "Type": "Put", "ColumnName": "attr_0", "Value": "hello_table_store", "Timestamp": 1506416585741 }, { "Type": "Put", "ColumnName": "attr_1", "Value": 1506416585881590900, "Timestamp": 1506416585741 } ] } ] }
Online debugging
Function Compute supports online debugging for functions. You can construct an event to trigger the function and test whether the function logic is as expected.
Tablestore events that trigger Function Compute are in the CBOR format, which is a JSON-like binary format. Therefore, you can perform online debugging in the following methods: