Use Function Compute - Tablestore - Alibaba Cloud Documentation Center

This topic describes how to use Function Compute to perform real-time computing on incremental data in Tablestore.

Background information

Function Compute: a fully-managed, event-driven computing service. It allows you to focus on coding without the need to procure and manage infrastructure resources such as servers. You need to only upload your code or image. Function Compute allocates computing resources, runs tasks in an elastic and reliable manner, and provides features such as log query, performance monitoring, and alerting. For more information, see What is Function Compute?

A Tablestore Stream is a tunnel that is used to obtain incremental data from Tablestore tables. After you create Tablestore triggers, a Tablestore Stream and a function in Function Compute can be automatically connected. This allows the custom program logic in the function to automatically process data modifications in Tablestore tables.

Scenarios

The following figure shows the tasks that can be executed by Function Compute.

Data synchronization: You can use Function Compute to synchronize real-time data that is stored in Tablestore to data cache, search engines, or other database instances.
Data archiving: You can use Function Compute to archive incremental data that is stored in Tablestore to OSS for cold backup.
Event-driven application: You can create triggers to trigger functions to call API operations that are provided by IoT Hub and cloud applications. You can also create triggers to send notifications.

Prerequisites

A Tablestore instance and a data table are created. For more information, see Create an instance and Create a data table.
Function Compute is activated, and a function is created. For more information, see Activate Function Compute and Create a service.

Usage notes

Make sure that the Tablestore data table resides in the same region as Function Compute.

Step 1: Enable the Stream feature for the data table

Before you create a trigger, you must enable the Stream feature for the data table in the Tablestore console to allow the function to process incremental data that is written to the table.

Log on to the Tablestore console.
In the top navigation bar, select a region.
On the Overview page, click the name of the instance that you want to manage or click Manage Instance in the Actions column.
In the Tables section of the Instance Details tab, click the name of the required data table and then click the Tunnels tab. Alternatively, you can click the icon and then click Tunnels.
On the Tunnels tab, click Enable in the Stream Information section.
In the Enable Stream dialog box, configure the Log Expiration Time parameter and click Enable.
The value of the Log Expiration Time parameter must be a non-zero integer and cannot be changed after it is specified. Unit: hours. Maximum value: 168.
Important
The Log Expiration Time parameter cannot be modified after it is set. Proceed with caution.

Step 2: Configure a Tablestore trigger

You can create a Tablestore trigger in the Function Compute console to process the real-time data stream in a Tablestore data table.

Note

To create a function in Tablestore, you need to select the Use Built-in Runtime option.

Go to the Functions tab.
1. Log on to the Function Compute console.
2. In the left-side navigation pane, click Services & Functions.
3. In the top navigation bar, select a region.
4. On the Services page, click the name of the service that you want to manage.
On the Functions tab that appears, click Create Function.

On the Create Function page, configure the parameters.

Select Use Built-in Runtime to create the function.
In the Basic Settings section, set Function Name, and set Handler Type to Event Handler.
Event Handler indicates that the execution of the function is triggered by timers, SDKs, APIs and triggers of other Alibaba Cloud services.
Note
If you do not specify a function name, Function Compute automatically generates a function name.

The following table describes how to configure the parameters in the Code section.

Parameter

Description

Runtime

Select a language, for example, Python, Java, PHP, or Node.js. In this example, Python 3.9 is selected.

For more information, see Runtimes supported by Function Compute.

Code Upload Method

The method used to upload the code to Function Compute. Valid values:

Use Sample Code (default value): Select sample codes provided by Function Compute to create a function based on your business requirements.
Upload ZIP: Select a ZIP package to upload your code.
Upload Folder: Select a folder that contains your code.
OSS: Specify the Bucket Name and Object Name parameters to upload your code.

In this case, Use Sample Code is selected. In the sample code list, select Trigger a Function by Tablestore.

The following table describes how to configure the parameters in the Advanced Settings section.

Parameter	Description
Specifications	Select or manually enter a combination of vCPU Capacity and Memory Capacity. For more information about the billing of Function Compute, see Billing overview. Important The ratio of vCPU capacity to memory capacity (in GB) ranges from 1:1 to 1:4.
Size of Temporary Disk	The disk used to temporarily store files of function instances. The temporary disk is attached to the root directory of the instance. Specify the size of the disk to temporarily store files. Valid values: 512 MB(Default Value): Disks of this specification are not billed. Function Compute provides free disks with a storage capacity of up to 512 MB. 10 GB: You are charged for disk usage of 9.5 GB. Important The lifecycle of a temporary disk is consistent with that of the function instance. When the instance is reclaimed by the system, the data on the disk is cleared.
Instance Concurrency	The number of requests a single function instance can process at a time. For more information, see Configure instance concurrency.
Execution Timeout Period	The time-out period for executing the function. The default value is 60 seconds, and the maximum value is 86400 seconds. If the function fails to be executed within this period of time, Function Compute returns a time-out error.
Handler	Function Compute loads and invokes the handler to process requests at run time.
Time zone	By default, the runtime environment of Function Compute uses the UTC time. You can configure the TZ environment variables to modify the time zone. When you select a time zone, Function Compute automatically adds a TZ environment variable to the function. The value of the TZ environment variable is the time zone that you set.

Optional: In the Environment Variables section, configure the variables for the runtime environment of the function. For more information, see Environment variables.

In the Trigger Configurations section, configure the parameters. The following table describes these parameters.

Parameter	Description
Trigger type	Select Tablestore.
Name	Enter a name for the trigger.
Instance	Select a Tablestore instance from the drop-down list.
Table	Select a data table from the drop-down list.
Role Name	Select AliyunTableStoreStreamNotificationRole. Note If this is the first time you create a trigger of this type, click Authorize Now in the message that appears after you click Create, and specify role details and permissions on the page that appears.

Click Create.
The trigger that you create is displayed on the Triggers tab.
Note
You can also view and create Tablestore triggers on the Trigger tab of the table in the Tablestore console.

Step 3: Test the function

Function Compute allows you to debug functions online. You can create an event to trigger a function and test whether the function logic is implemented as expected.

Tablestore events that trigger Function Compute are in the CBOR format, which is a JSON-like binary format. You can debug a function online by performing the following steps:

Write code.

On the Functions tab, click the name of the function that you want to manage.

On the function details page, click the Code tab to write code in the code editor.

Note

In this example, the function code is written in Python.

In the code, records = json.loads(event) is used to process custom trigger events for the test.

import logging
import cbor
import json
def get_attribute_value(record, column):
    attrs = record[u'Columns']
    for x in attrs:
        if x[u'ColumnName'] == column:
            return x['Value']
def get_pk_value(record, column):
    attrs = record[u'PrimaryKey']
    for x in attrs:
        if x['ColumnName'] == column:
            return x['Value']
def handler(event, context):
    logger = logging.getLogger()
    logger.info("Begin to handle event")
    #records = cbor.loads(event)
    records = json.loads(event)
    for record in records['Records']:
        logger.info("Handle record: %s", record)
        pk_0 = get_pk_value(record, "pk_0")
        attr_0 = get_attribute_value(record, "attr_0")
    return 'OK'

Test the function.
1. On the Code tab, click the icon next to Test Function and select Configure Test Parameters from the drop-down list.
2. In the Configure Test Parameters panel, click the Create New Test Event tab. Select Tablestore from the Event Template drop-down list and enter the event name and event content. Click OK.
  Note
  You can select the name of a created event on the Modify Existing Test Event tab.
  For more information about the data format of event content, see Appendix: Data processing.
3. Click Test Function to check whether the function performs as expected.
Modify and deploy the code.
1. If OK is returned after records=json.loads(event) is executed, set records to records.
2. Click Deploy.
When data is written to Tablestore, the function logic is triggered.

Appendix: Data processing

When you configure test parameters, you must specify the event content based on a specific data format.

Data format

A Tablestore trigger encodes incremental data in the CBOR format to create a Function Compute event. The following code provides an example of the format of incremental data:

{
    "Version": "string",
    "Records": [
        {
            "Type": "string",
            "Info": {
                "Timestamp": int64
            },
            "PrimaryKey": [
                {
                    "ColumnName": "string",
                    "Value": formated_value
                }
            ],
            "Columns": [
                {
                    "Type": "string",
                    "ColumnName": "string",
                    "Value": formated_value,
                    "Timestamp": int64
                }
            ]
        }
    ]
}

Parameters

The following table describes the parameters that are included in the preceding format.

Parameter	Description
Version	The version of the payload, which is Sync-v1. Data type: string.
Records	The array that stores the row of incremental data in the data table. This parameter includes the following fields: Type: the type of the operation that is performed on the row. Valid values: PutRow, UpdateRow, and DeleteRow. Data type: string. Info: includes the Timestamp field, which specifies the time when the row was last modified. The value of Timestamp must be in UTC. Data type: int64.
PrimaryKey	The array that stores the primary key column. This parameter includes the following fields: ColumnName: the name of the primary key column. Data type: string. Value: the content of the primary key column. Data type: formated_value. Valid values: integer, string, and blob.
Columns	The array that stores the attribute columns. This parameter includes the following fields: Type: the type of the operation that is performed on the attribute column. Valid values: Put, DeleteOneVersion, and DeleteAllVersions. Data type: string. ColumnName: the name of the attribute column. Data type: string. Value: the content of the attribute column. Data type: formated_value. Valid values: integer, boolean, double, string, and blob. Timestamp: the time when the attribute column was last modified. The value of Timestamp must be in UTC. Data type: int64.

Data examples

{
    "Version": "Sync-v1",
    "Records": [
        {
            "Type": "PutRow",
            "Info": {
                "Timestamp": 1506416585740836
            },
            "PrimaryKey": [
                {
                    "ColumnName": "pk_0",
                    "Value": 1506416585881590900
                },
                {
                    "ColumnName": "pk_1",
                    "Value": "2017-09-26 17:03:05.8815909 +0800 CST"
                },
                {
                    "ColumnName": "pk_2",
                    "Value": 1506416585741000
                }
            ],
            "Columns": [
                {
                    "Type": "Put",
                    "ColumnName": "attr_0",
                    "Value": "hello_table_store",
                    "Timestamp": 1506416585741
                },
                {
                    "Type": "Put",
                    "ColumnName": "attr_1",
                    "Value": 1506416585881590900,
                    "Timestamp": 1506416585741
                }
            ]
        }
    ]
}

Appendix: Create a trigger by using an existing Function Compute service and function

You can use an existing Function Compute function and service to create a Tablestore trigger in the Tablestore console.

Log on to the Tablestore console.
On the Overview page, click the name of the instance that you want to manage or click Manage Instance in the Actions column.
In the Tables section of the Instance Details tab, click the name of the required data table.
On the Trigger tab, click Use Existing Function Compute.
In the Create Trigger dialog box, select a Function Compute service for FC Service and a Function Compute function for FC Function, enter a trigger name, and then click OK.